A Redis Based Background job Processor for Deno

TaskerMQ

TaskerMQ is a Redis based simple background job processor for Deno 🦕.

Usage

// sampleWorker.ts
import { WorkerClass } from 'https://deno.land/x/taskermq/mod.ts';

export default class SampleWorker extends WorkerClass {
  // this is the function you need to perform the background job
  perform(params) {
    console.log(params[0] + params[1])
  }
}
// work.ts
import { Connection, Tasker } from "https://deno.land/x/taskermq/mod.ts";
import SampleWorker from "./samplerWorker.ts";

// Configure your connection to the redis instance
const connection = new Connection({
  hostname: "localhost",
  port: 6379,
});

// Accept a new queue name for your task
const queue = Deno.args[0] || "taskerMQdefault";

// Connect to the redis instance
await connection.connectRedis();

// Create a new Tasker instance using the redis connection, queue name and an array of worker classes
const tasker = new Tasker(connection, queue, [SampleWorker]);

// Run tasker as a long running process.
tasker.run();

Our job processor is now set-up. Now we have to start adding jobs to the queue. TaskerMQ assumes the jobs to be in serialized JSON with the following format:

{"klass": "SampleWorker", "params": [1, 2]}

TaskerMQ will try to deserialize the string and execute the job.

NOTE: Any job not in the correct format will be ignored and removed from the queue.

To run the worker:

deno run --allow-net work.ts sample-worker-queue

Download Details:

Author: ronyv89

Source Code: https://github.com/ronyv89/taskermq

#deno #node #nodejs #javascript

What is GEEK

Buddha Community

A Redis Based Background job Processor for Deno
Loma  Baumbach

Loma Baumbach

1596679140

Redis Transactions & Long-Running Lua Scripts

Redis offers two mechanisms for handling transactions – MULTI/EXEC based transactions and Lua scripts evaluation. Redis Lua scripting is the recommended approach and is fairly popular in usage.

Our Redis™ customers who have Lua scripts deployed often report this error – “BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE”. In this post, we will explain the Redis transactional property of scripts, what this error is about, and why we must be extra careful about it on Sentinel-managed systems that can failover.

Redis Lua Scripts Diagram - ScaleGrid Blog

Transactional Nature of Redis Lua Scripts

Redis “transactions” aren’t really transactions as understood conventionally – in case of errors, there is no rollback of writes made by the script.

Atomicity” of Redis scripts is guaranteed in the following manner:

  • Once a script begins executing, all other commands/scripts are blocked until the script completes. So, other clients either see the changes made by the script or they don’t. This is because they can only execute either before the script or after the script.
  • However, Redis doesn’t do rollbacks, so on an error within a script, any changes already made by the script will be retained and future commands/scripts will see those partial changes.
  • Since all other clients are blocked while the script executes, it is critical that the script is well-behaved and finishes in time.

The ‘lua-time-limit’ Value

It is highly recommended that the script complete within a time limit. Redis enforces this in a weak manner with the ‘lua-time-limit’ value. This is the maximum allowed time (in ms) that the script is allowed to run. The default value is 5 seconds. This is a really long time for CPU-bound activity (scripts have limited access and can’t run commands that access the disk).

However, the script is not killed when it executes beyond this time. Redis starts accepting client commands again, but responds to them with a BUSY error.

If you must kill the script at this point, there are two options available:

  • SCRIPT KILL command can be used to stop a script that hasn’t yet done any writes.
  • If the script has already performed writes to the server and must still be killed, use the SHUTDOWN NOSAVE to shutdown the server completely.

It is usually better to just wait for the script to complete its operation. The complete information on methods to kill the script execution and related behavior are available in the documentation.

#cloud #database #developer #high availability #howto #redis #scalegrid #lua-time-limit #redis diagram #redis master #redis scripts #redis sentinel #redis servers #redis transactions #sentinel-managed #server failures

A Redis Based Background job Processor for Deno

TaskerMQ

TaskerMQ is a Redis based simple background job processor for Deno 🦕.

Usage

// sampleWorker.ts
import { WorkerClass } from 'https://deno.land/x/taskermq/mod.ts';

export default class SampleWorker extends WorkerClass {
  // this is the function you need to perform the background job
  perform(params) {
    console.log(params[0] + params[1])
  }
}
// work.ts
import { Connection, Tasker } from "https://deno.land/x/taskermq/mod.ts";
import SampleWorker from "./samplerWorker.ts";

// Configure your connection to the redis instance
const connection = new Connection({
  hostname: "localhost",
  port: 6379,
});

// Accept a new queue name for your task
const queue = Deno.args[0] || "taskerMQdefault";

// Connect to the redis instance
await connection.connectRedis();

// Create a new Tasker instance using the redis connection, queue name and an array of worker classes
const tasker = new Tasker(connection, queue, [SampleWorker]);

// Run tasker as a long running process.
tasker.run();

Our job processor is now set-up. Now we have to start adding jobs to the queue. TaskerMQ assumes the jobs to be in serialized JSON with the following format:

{"klass": "SampleWorker", "params": [1, 2]}

TaskerMQ will try to deserialize the string and execute the job.

NOTE: Any job not in the correct format will be ignored and removed from the queue.

To run the worker:

deno run --allow-net work.ts sample-worker-queue

Download Details:

Author: ronyv89

Source Code: https://github.com/ronyv89/taskermq

#deno #node #nodejs #javascript

Gerhard  Brink

Gerhard Brink

1620692100

10 Latest Big Data Engineer Openings At Top Firms In India

Extras:

1| Senior Technical Architect at Thoucentric

Location: Bangalore

**Responsibilities: **

  • Design and implement data architecture and ETL for a niche data platform.
  • Bring in-depth understanding on Relational, Big Data and Cloud technologies.
  • Build client relationships and participate in business development and proposal work to grow a strong data engineering sub-practice.

Apply here.

2| Data Engineer at Thoucentric

Location: Bangalore

**Responsibilities: **

  • Build data crawlers to extract data from customers’ data sources using available ETL platforms, and troubleshoot the issues faced during data loading & processing.
  • Design and build data warehouse models in columnar databases.
  • Develop data processing scripts using SQL and optimise complex sequences of SQL Queries.

Apply here.

3| Big Data Engineer at Thoucentric

Location: Bangalore

Responsibilities:

  • Take ownership of end-to-end data-pipeline including system design and integrating required Big Data tools & frameworks.
  • Implementing ETL processes and constructing data warehouse (HDFS, S3, Azure etc.) at scale.
  • Analyse the source and target system data. Map the transformation that meets the requirements.

Apply here.

Find below the data engineer job openings:

#careers #aim weekly job alerts #aimrecruits #big data engineer jobs at top firms #big data engineers job #big data jobs #data science jobs #top firm data science jobs #weekly job openings list

15 Latest Data Science And Analyst Jobs To Apply For

For this week’s latest data science job openings, we have come up with a curated list of job openings for data scientists and analysts from last week.

Data Scientists Openings

Data Science Lead at Paytm Money

Location: Bangalore

Responsibilities:

#data science job search #data science jobs #data science jobs in india #latest data science job openings #latest data science jobs #onboarding data science jobs

Bella Garvin

Bella Garvin

1624161311

Job Portal Development Company

We at Orbit Edge are a well-noted and trusted job portal development company. We create applications that are simple to learn and simple to use. Our committee of job portal application developers is passionate about providing complex solutions for job portal apps. They are experts in providing effective and robust applications to match your business necessities.

#job portal development #job portal app development #online job-portal development #job portal development company #job portal application development