BeginnersShellScripting for BatchJobs

EvanBollig GeoffreyWomeldorff
and

Beforewebegin...
Everyonepleasevisitthispageforexample scriptsandgrabacribsheetfromthefront
[Link]

Alsopleasesigninontheclipboardbeing passedaround

Outline
IntroductiontoShells
History Whatisthe*nixENV? WhatisShellScripting?

NoviceScripting
HelloWorld([Link]) Variables([Link]) `...`,'...'and...([Link])

 IntermediateScripting

Outline(2)

FileandOutputControl([Link]) Loops,ConditionalsandUtilizingAvailablePrograms ([Link])

AdvancedScripting
SubstringsandCaseStatementsforMachineandArchitecture SpecificTasks([Link]) RestartFiles,IntegersandAvoidingRedundantTasks ([Link]) CallingMatlabforBatchProcessing([Link])

IntroductiontoShells

Whatare*nixShells?
UnixwasthefirstOStoseparatethecommand interpreter(shell)fromtheoperatingsystem(kernel) Shellsabstractthecomplexitiesofsystemsand providestandardcommandsforallmachines Ashellhasbuiltincommands,plusallowsusaccess tootherUnixcommands
history,export,cd>builtin cp,ls,grep>Unixcommands(/bin,/usr/bin,etc.)

[Link]

AGeneralHistory
BourneShell(/bin/sh):
BellLaboratories,byStephenBourne Basisforallshells,butprovidedlimitedfunctionality

CShell(/bin/csh):
UCBerkley,byBillJoy(creatorofvi) ExpandedBourneShelltomimicClanguageandprovideinteractivity(in reality,itwastoobuggytouseforscripts) TENEXcsh(/bin/tcsh)waslaterdevelopedtogetridofbugsbutUnix manufacturersstuckwithcsh

[Link]

History(2)
KornShell(/bin/ksh):
AT&T,byDavidKorn Hybridofshandcshusingshlanguagewithcshinteractivity Althoughitwasthebestofbothworlds,itwasstillfromAT&T=>ksh wasnotfree.

BourneAgainShell(/bin/bash):
GNU'sPOSIXcompatibleBourneShell EquivalenttoKornbutfreeimplementation PerfectfitinLinuxwhichresultedinmasspopularity

rc,zshandothersarestillemerging...

UnderstandingtheENV
TheENVistheUnixenvironment
Key=valuepairsdescribecurrentconfiguration Everyrunningapplication,user,etcthatinteractswiththe operatingsystemhasanenvironment Aprocessinheritsitsparent'senvironmentunlesstheparent overrides/specifiesotherproperties. Users'initialENVisprovidedbytheshellandbasedon /etc/profile.d/*,/etc/*shrcand~/.*shrcfiles. Changestochild'senvironmentonlyaffectchild'schildren andarelostwhenthesubtreeexits.

ControllingtheEnv
Shellsallowustocontroltheenvironmentwithbuiltin commands;Unixcommandscanalsobeused: %>env
Listsyourcurrentenvironment

%>export<key>=<value>
Setanewkeyorupdateanexistingone NOTE:thisisequivalenttosetenv<key><value>intheCshell family

%>echo$<key>
Printcurrentvalueofvariable

UnderstandingShellScripts
Shellscriptsarefilesthatcontaincommandsasyou wouldrunthemonthecommandprompt. Scriptscommonlyusevariables,conditionalsandloops.
Theseareallavailableonthecommandprompt

Whatmakesascriptsospecial? Ithastheexecutablebitset(chmod+x<script>) Itexecutesuntilitencountersanerror(interpreted) Scriptscanuseanyavailableexecutables;noneedto worryaboutlibraries,headersorAPIs

WhyScript?
Quickanddirtysolutions
Mostlyyouusepreexistingbinaries Changestoscripthaveimmediateeffects noneedtowrite/(re)compilecode

Interprogramcommunicationisperformedthrough standardfileI/Opipes
Noworriesaboutcommunicationstandards ShellScriptsuseshellsthatarestandardonall*nixplatforms NOTE:[Link] forthecommandsascriptcontains.

NoviceShellScripting

HelloWorld
Example1:[Link] Keyfeatures:
Shbang(#!/bin/sh)
Everyshellscriptmuststartwithashbangonthefirst [Link] interpretit([Link],bash,csh,perl,python,tcl,etc.)

Comments
Commentsbeginwiththehash(#)andproceedtotheend oftheline

Variables
Example2:[Link];[Link] KeyFeatures:
LocalandGlobalVariables
<key>=<value>isalocalvariablethatisnotpassed tochildren export<key>=<value>isaglobalvariablethatis passedtochildrenbutisnotvisibletoparentofthis script

`...`,'...'and... Example3:[Link] KeyFeatures:

IntermediateShellScripting

FileandOutputControl
Example4:[Link];[Link] KeyFeatures:
cat<<EOF...
Printeverythingbetween<<EOFandthenextoccurrenceof EOFtostdout. NOTE:the>[Link]<<EOFredirects stdouttoafile

Redirectingbothstdoutandstderr
Use([command]>[stdoutFile])>[stderrFile]tostoreoutput separately

Use[command]&>[outFile]tostoretogether

Loops,ConditionalsandUtilizingAvailable Programs
Example5:[Link] KeyFeatures:
Forloopexecutesonlistoffilenames Conditional(ifthenelse)testsforalockfiletoavoidextra work WetakeadvantageofstandardUnixcommandslikemkdir, pwd,grepandechobutalsousenonstandard Imagemagicktoperformbatchimageconversion

AdvancedShellScripting

SubstringsandCaseStatementsforMachine andArchitectureSpecificTasks
Example6:[Link] KeyFeatures:
Casestatementtocomparemachine'sname(hostname) Casestatementtocomparearchitectureofmachine Substringsfromvariablesusing${var_name#*/}and ${var_name%/*}

RestartFiles,IntegersandAvoiding RedundantTasks
Example7:[Link] KeyFeatures:
Restartfileinformsscriptwhattasksithasalready completed Integersrequireuniquehandling(wecannotcompare likestrings) LocatesALLjpegsinALLsubdirectoriesandrebuilds subdirsinoutputfolder(importanttokeepbatchjobs organized)

CallingMatlabasforBatchProcessing
Example8:[Link];testMatlab.m; mandrill_cvt.m KeyFeatures:
Permachinetasks
differentcompressionforeachcluster:Class*,Prism*, Hallway*,andVislabmachines

ExecutesMatlabbehindthesceneswithacomplex clusteringalgorithmfordatacompression(easilycoded inMatlabandeasytovisualizeresults)

Conclusion
Younowhaveexampleshellscriptstogetstarted. UsetheresourcesonthecribsheetandontheSCS TechSupportTwikitohelpyouwriteyourownscripts. Uselogging,locksandrestartfilestohelpaccelerate batchjobs. Remember:don'treinventthewheel!Ifacommandexists anditfunctionscorrectly,don'twasteyourtimerewriting [Link]...

SpecialThanks...
toTSGfortheexcusetoplaywithshellscripts toGordonErlebacherfortheextratimeallotment awayfromWATT toSCSCustodialformakinglifeinterestingduringthe weehoursofthemorning

Andofcourse!Thankyouallforattending...