locked
Options for batch processing RRS feed

  • Question

  • Hi,

    I'm seeking input and advice for options for setting up and running complex batch jobs. Currently we have numerous console apps and use the Task Scheduler. This is creaking now and we need something better, easy to configure and adjust, easy to arbitrarily pause running jobs or pause the scheduling, robust status and error reporting, flexible dependencies, retries, and so on.

    I saw a company named JAMS who seem to know their stuff but do not quote prices.

    Is there really not much out there to handle this in a "mainframe" like reliable manner?

    It would be nice to establish a solid pattern/template for a "job" (this could just be a console with a bunch of standard stuff for arg processing and logging etc) so that writing new ones requires little effort from the developer (other than adding the business logic).

    Quartz.NET looks interesting but I haven't had time to evaluate it.

    Thx




    Saturday, May 5, 2018 2:53 PM

All replies

  • Yes, mainframe programmers say "batch" but I can't really define what we meant by that. I don't remember it referring to batches, as in transaction batches. Also, just what are jobs? I know that they are multiple "exec" statements, each with multiple "DD" statements but you can't say that here, the vast majority of members here don't understand.

    You need to think about what you need to do in terms of requirements, not the terms (words) we used to use. Windows does not have the equivalent of DD statements so instead of saying DD statement we must say something like:

    Something to configure the input and output.

    It depends on the language used. If the programs are developed using COBOL then I assume you have files in the environment so you need the equivalent of a DD statement. Other users of the language would provide the best help with that. Otherwise I assume you don't need the equivalent of a DD statement.

    A Windows batch file (bat or cmd extension) is the closest equivalent to a MVS job. However you can do everything they can do using VBScript or JavaScript. You can use the Task Scheduler to schedule a script to run.

    In terms of pausing, arg processing and logs you need to describe what that means in terms of your requirements. You could say you need to do things the same way that things worked in the past or you could describe the requirements and look for solutions that are better than the past.



    Sam Hobbs
    SimpleSamples.Info


    Sunday, May 6, 2018 6:47 PM
  • Yes, mainframe programmers say "batch" but I can't really define what we meant by that. I don't remember it referring to batches, as in transaction batches. Also, just what are jobs? I know that they are multiple "exec" statements, each with multiple "DD" statements but you can't say that here, the vast majority of members here don't understand.

    You need to think about what you need to do in terms of requirements, not the terms (words) we used to use. Windows does not have the equivalent of DD statements so instead of saying DD statement we must say something like:

    Something to configure the input and output.

    It depends on the language used. If the programs are developed using COBOL then I assume you have files in the environment so you need the equivalent of a DD statement. Other users of the language would provide the best help with that. Otherwise I assume you don't need the equivalent of a DD statement.

    A Windows batch file (bat or cmd extension) is the closest equivalent to a MVS job. However you can do everything they can do using VBScript or JavaScript. You can use the Task Scheduler to schedule a script to run.

    In terms of pausing, arg processing and logs you need to describe what that means in terms of your requirements. You could say you need to do things the same way that things worked in the past or you could describe the requirements and look for solutions that are better than the past.



    Sam Hobbs
    SimpleSamples.Info


    Yes the terms "batch" are misused both where I work and many other places.

    The actual nature of the processing is something with these overall characteristics

    1. Has no UI to speak of.

    2. Runs usually based on a time of day or as the result of some event like another "batch" app ending.

    3. Performs repetivive processing of sets of data (e.g. trawling through a database for all users added today)

    4. Is not a Web app, a Windows forms/wpf app or web service.

    So these are usually implemented as .Net Console apps and run by (say) the Task Scheduler.

    What I'm seeking is a solid design pattern and framework for doing these kinds of things, Console apps might be OK but it would be neat of the processing were some form of "package" or assembly that implements some form of interface and then have a general purpose "job execution engine" that can load and run these.

    I guess what I'm saying is that there seems to little or no support for this kind of data processing on Windows - or else I'm just not that well informed!

    Monday, May 7, 2018 8:20 PM
  • You are still being vague about details.

    Do you mean something like CA Workload Automation CA 7 Edition - CA Technologies? That is of course much more complex than what you need.

    Have you been to IBM MAINFRAME & MVS FORUMS? You should be able to post a question somewhere there to ask if a Windows equivalent of something you are familiar with exists.



    Sam Hobbs
    SimpleSamples.Info

    Monday, May 7, 2018 9:49 PM
  • You are still being vague about details.

    Do you mean something like CA Workload Automation CA 7 Edition - CA Technologies? That is of course much more complex than what you need.

    Have you been to IBM MAINFRAME & MVS FORUMS? You should be able to post a question somewhere there to ask if a Windows equivalent of something you are familiar with exists.



    Sam Hobbs
    SimpleSamples.Info

    I don't know what more details I can provide you with, this is nothing to do with mainframes either but Windows.

    Currently we have a bunch of .Net console apps which are started during the night by task scheduler on one of our servers.

    These do stuff like (for example) pulling new records added to the DB that day and generating emails and updating these records to flag them as "EmailSent" this is the kind of thing, repetitive data processing.

    These run for like a couple of hours and they're setup to run in some order too.

    One of them also pulls a data file from FTP and then uses that to update potentially thousands of rows in the DB.

    These are referred to by staff as "the overnight batch jobs".

    That's it, so I'm seeking alternative ways of implementing this (i.e. not using crude console apps and not relying on task scheduler) I simply don't know what larger organizations do these days for this kind of processing.

    Thx


    Sunday, May 20, 2018 5:29 PM