brief introduction
In development, we may encounter the need to invoke scripts in the program, or involve the interaction between two languages. The author met the need to call Python in go before, and then applied the go-python3 Library in the code. Actually, calling the script of Python in go is also a solution. This article will introduce the method of running shell script in go and the corresponding analysis of its source code.
Program use case
test_command.go
package learn
import (
"fmt"
"os/exec"
"testing"
)
func TestCmd(t *testing.T) {
if o, e := exec.Command("./test.sh", "1", "2").Output(); e != nil {
fmt.Println(e)
} else {
fmt.Println(string(o))
}
}test.sh
#!/bin/bash a=$1 b=$2 echo $a echo $b
The above example means to run test SH this script, and the input parameters are 1 and 2. What is written in the script is relatively simple, that is, printing these two input parameters. In fact, the key to the problem is exec Command() method, let's get to the bottom and find out.
Source code analysis
func Command(name string, arg ...string) *Cmd {
cmd := &Cmd{
Path: name,
Args: append([]string{name}, arg...),
}
if filepath.Base(name) == name {
if lp, err := LookPath(name); err != nil {
cmd.lookPathErr = err
} else {
cmd.Path = lp
}
}
return cmd
}
// Base returns the last element of the path.
// The trailing path separator is removed before the last element is extracted.
// If the path is empty, Base returns ".".
// If the path consists entirely of separators, Base returns a single separator.
func Base(path string) string {
if path == "" {
return "."
}
// Strip trailing slashes.
for len(path) > 0 && os.IsPathSeparator(path[len(path)-1]) {
path = path[0 : len(path)-1]
}
// Throw away volume name
path = path[len(VolumeName(path)):]
// Find the last element
i := len(path) - 1
for i >= 0 && !os.IsPathSeparator(path[i]) {
i--
}
if i >= 0 {
path = path[i+1:]
}
// If empty now, it had only slashes.
if path == "" {
return string(Separator)
}
return path
}
//LookPath searches the directory named by the PATH environment variable for an executable file named file input parameter. If the file contains a slash, it will try directly without referring to PATH. The result may be an absolute PATH or a PATH relative to the current directory.
func LookPath(file string) (string, error) {
if strings.Contains(file, "/") {
err := findExecutable(file)
if err == nil {
return file, nil
}
return "", &Error{file, err}
}
path := os.Getenv("PATH")
for _, dir := range filepath.SplitList(path) {
if dir == "" {
// Unix shell semantics: path element "" means "."
dir = "."
}
path := filepath.Join(dir, file)
if err := findExecutable(path); err == nil {
return path, nil
}
}
return "", &Error{file, ErrNotFound}
}
// Find an executable command with the same name as file
func findExecutable(file string) error {
d, err := os.Stat(file)
if err != nil {
return err
}
if m := d.Mode(); !m.IsDir() && m&0111 != 0 {
return nil
}
return os.ErrPermission
}Through the above description of exec From the analysis of the command () source code, we can know that this function just looks for the executable file with the same name as the path and constructs a Cmd object return. It is worth noting here that if the path we enter is not the specific path of an executable file, we will go to the registered path in the path environment variable to find a command with the same name as path. If it is not found at this time, an error will be reported.
Then let's take a look at what is sacred, what is the use, and how to use it. Let's take a look at what's in this structure.
// The Cmd structure represents an external command that is ready or executing
// A Cmd object cannot be reused after the Run, Output or CombinedOutput methods are called.
type Cmd struct {
// Path represents the path to run the command
// This field is the only one that needs to be assigned. It cannot be an empty string,
// And if Path is a relative Path, the directory pointed to by the Dir field is referenced
Path string
// The Args field represents the parameters required to call the command, where Path exists in the form of Args[0] when running the command
// If this parameter is empty, run the command directly using Path
//
// In common scenarios, both Path and Args parameters are used when calling commands
Args []string
// Env represents the environment variable of the current process
// The entries in each Env array exist in the form of "key=value"
// If Env is nil, the process created by running the command there will use the environment variable of the current process
// If there is a duplicate key in Env, the last value in the key will be used.
// There are special cases in Windows. If SYSTEMROOT is missing in the system, or the environment variable is not set to an empty string, its operations are append operations.
Env []string
// Dir represents the running path of the command
// If Dir is an empty string, the command will run in the running path of the current process
Dir string
// Stdin represents the standard input stream of the system
// If Stdin is a * OS File, the standard input of the process will be directly connected to the file.
Stdin io.Reader
// Stdout represents the standard output stream
// If StdOut is a * OS File, the standard input of the process will be directly connected to the file.
// It is worth noting that if StdOut and StdErr are the same object, only one coroutine can call Writer at the same time
Stdout io.Writer
Stderr io.Writer
// ExtraFiles specifies additional open files inherited by the new process. It does not include standard input, standard output, or standard error. If it is not zero, item i becomes file descriptor 3+i.
// The first three elements of ExtraFiles are stdin, stdout and stderr
// ExtraFiles is not supported on Windows
ExtraFiles []*os.File
SysProcAttr *syscall.SysProcAttr
// When the command runs, Process is the Process represented by the command
Process *os.Process
// ProcessState contains information about an exiting process and is available after calling Wait or Run.
ProcessState *os.ProcessState
ctx context.Context // ctx can be used for timeout control
lookPathErr error // If an error occurs when calling LookPath to find the path, it is assigned to this field
finished bool // When Wait is called once, it will be set to True to prevent repeated calls
childFiles []*os.File
closeAfterStart []io.Closer
closeAfterWait []io.Closer
goroutine []func() error //A series of functions, which will be executed together when calling Satrt to start executing the command. Each function is assigned a goroutine to execute
errch chan error // It is used in conjunction with the previous field. Through this chan, the execution result of the above function is transmitted to the current goroutine
waitDone chan struct{}
}Above, we have analyzed some fields of the Cmd structure. It can be understood that Cmd is an abstraction within the life cycle of a command. Let's analyze the method of Cmd and see how it is used.
// The Run method starts executing the command and waits for it to finish running
// If the command runs, there is no problem copying stdin, stdout, and stder, and exits with a zero exit state, the error returned is nil.
// If the command starts but does not complete successfully, the error type is * exitrerror. In other cases, other error types may be returned.
// If the called goroutine has been used with runtime Lockosthread locks the operating system thread and modifies any inheritable OS level thread state (for example, Linux or Plan 9 namespace). The new process will inherit the caller's thread state.
func (c *Cmd) Run() error {
if err := c.Start(); err != nil {
return err
}
return c.Wait()
}
// The Start method starts the specified command, but does not wait for it to complete.
//
// If Start returns successfully, the c.Process field will be set.
//
// Once the command runs, the Wait method returns the exit code and releases the related resources.
func (c *Cmd) Start() error {
if c.lookPathErr != nil {
c.closeDescriptors(c.closeAfterStart)
c.closeDescriptors(c.closeAfterWait)
return c.lookPathErr
}
if runtime.GOOS == "windows" {
lp, err := lookExtensions(c.Path, c.Dir)
if err != nil {
c.closeDescriptors(c.closeAfterStart)
c.closeDescriptors(c.closeAfterWait)
return err
}
c.Path = lp
}
if c.Process != nil {
return errors.New("exec: already started")
}
if c.ctx != nil {
select {
case <-c.ctx.Done():
c.closeDescriptors(c.closeAfterStart)
c.closeDescriptors(c.closeAfterWait)
return c.ctx.Err()
default:
}
}
//Initialize and populate ExtraFiles
c.childFiles = make([]*os.File, 0, 3+len(c.ExtraFiles))
type F func(*Cmd) (*os.File, error)
//Stdin, stdout and stderr methods will be called here. If stdin, stdout and stderr of Cmd are not nil, the relevant copy tasks will be encapsulated as func and placed in the goroutine field, waiting to be called when the Start method is executed.
for _, setupFd := range []F{(*Cmd).stdin, (*Cmd).stdout, (*Cmd).stderr} {
fd, err := setupFd(c)
if err != nil {
c.closeDescriptors(c.closeAfterStart)
c.closeDescriptors(c.closeAfterWait)
return err
}
c.childFiles = append(c.childFiles, fd)
}
c.childFiles = append(c.childFiles, c.ExtraFiles...)
// If the Env of cmd is not assigned, the environment variable of the current process is used
envv, err := c.envv()
if err != nil {
return err
}
// This command will be used to start a new process
// On the Linux system, Frok is called to create another process at the bottom. Due to the limited space of the article, we will not analyze it in detail here. For details, see extended reading
c.Process, err = os.StartProcess(c.Path, c.argv(), &os.ProcAttr{
Dir: c.Dir,
Files: c.childFiles,
Env: addCriticalEnv(dedupEnv(envv)),
Sys: c.SysProcAttr,
})
if err != nil {
c.closeDescriptors(c.closeAfterStart)
c.closeDescriptors(c.closeAfterWait)
return err
}
c.closeDescriptors(c.closeAfterStart)
// Chan will not be applied unless goroutine is to be started
if len(c.goroutine) > 0 {
c.errch = make(chan error, len(c.goroutine))
for _, fn := range c.goroutine {
go func(fn func() error) {
c.errch <- fn()
}(fn)
}
}
// Timeout control
if c.ctx != nil {
c.waitDone = make(chan struct{})
go func() {
select {
case <-c.ctx.Done(): //If it times out, Kill the process executing the command
c.Process.Kill()
case <-c.waitDone:
}
}()
}
return nil
}
func (c *Cmd) stdin() (f *os.File, err error) {
if c.Stdin == nil {
f, err = os.Open(os.DevNull)
if err != nil {
return
}
c.closeAfterStart = append(c.closeAfterStart, f)
return
}
if f, ok := c.Stdin.(*os.File); ok {
return f, nil
}
//Pipe returns a pair of connected Files; The data read from r returns the bytes written to w.
pr, pw, err := os.Pipe()
if err != nil {
return
}
c.closeAfterStart = append(c.closeAfterStart, pr)
c.closeAfterWait = append(c.closeAfterWait, pw)
//Add related tasks to goroutine
c.goroutine = append(c.goroutine, func() error {
_, err := io.Copy(pw, c.Stdin)
if skip := skipStdinCopyError; skip != nil && skip(err) {
err = nil
}
if err1 := pw.Close(); err == nil {
err = err1
}
return err
})
return pr, nil
}
func (c *Cmd) stdout() (f *os.File, err error) {
return c.writerDescriptor(c.Stdout)
}
func (c *Cmd) stderr() (f *os.File, err error) {
if c.Stderr != nil && interfaceEqual(c.Stderr, c.Stdout) {
return c.childFiles[1], nil
}
return c.writerDescriptor(c.Stderr)
}
func (c *Cmd) writerDescriptor(w io.Writer) (f *os.File, err error) {
if w == nil {
f, err = os.OpenFile(os.DevNull, os.O_WRONLY, 0)
if err != nil {
return
}
c.closeAfterStart = append(c.closeAfterStart, f)
return
}
if f, ok := w.(*os.File); ok {
return f, nil
}
pr, pw, err := os.Pipe()
if err != nil {
return
}
c.closeAfterStart = append(c.closeAfterStart, pw)
c.closeAfterWait = append(c.closeAfterWait, pr)
//Add related tasks to goroutine
c.goroutine = append(c.goroutine, func() error {
_, err := io.Copy(w, pr)
pr.Close() // in case io.Copy stopped due to write error
return err
})
return pw, nil
}
// Wait for the command to exit and wait for any replication to stdin or from stdout or stderr to complete.
// The Start method must be called before calling Wait
// If the command runs, there is no problem copying stdin, stdout, and stder, and exits with a zero exit state, the error returned is nil.
// If the command fails or does not complete successfully, the error type is * exitrerror. Other error types may be returned for I/O problems.
// If any of c.Stdin, c.Stdout, or c.Stderr is not * OS File and Wait also Wait for their respective I/O cycles to be copied to or from the process
//
// Wait releases any resources associated with Cmd.
func (c *Cmd) Wait() error {
if c.Process == nil {
return errors.New("exec: not started")
}
if c.finished {
return errors.New("exec: Wait was already called")
}
c.finished = true
//Wait for the process to finish running and exit
state, err := c.Process.Wait()
if c.waitDone != nil {
close(c.waitDone)
}
c.ProcessState = state
//Check the function above the goroutine field for errors
var copyError error
for range c.goroutine {
if err := <-c.errch; err != nil && copyError == nil {
copyError = err
}
}
c.closeDescriptors(c.closeAfterWait)
if err != nil {
return err
} else if !state.Success() {
return &ExitError{ProcessState: state}
}
return copyError
}
// Output runs the command and returns its standard output.
// Any returned error is usually of type * exitrerror.
// OutPut actually encapsulates the execution flow of the command and formulates the OutPut flow of the command
func (c *Cmd) Output() ([]byte, error) {
if c.Stdout != nil {
return nil, errors.New("exec: Stdout already set")
}
var stdout bytes.Buffer
c.Stdout = &stdout
captureErr := c.Stderr == nil
if captureErr {
c.Stderr = &prefixSuffixSaver{N: 32 << 10}
}
err := c.Run()
if err != nil && captureErr {
if ee, ok := err.(*ExitError); ok {
ee.Stderr = c.Stderr.(*prefixSuffixSaver).Bytes()
}
}
return stdout.Bytes(), err
}From the above method analysis, we can see that the process of running a command is run - > Start - > wait, waiting for the command to run. And at start, a new process will be up to execute the command. Based on the above analysis of Cmd, the author feels that the test code written at the beginning of the article is really poor, because Cmd encapsulates many things. We can make full use of its encapsulated functions in our work, such as setting timeout, setting standard input stream or standard output stream, You can also customize the environment variables for the execution of this command, and so on....
Extended reading
- About fork and exec: www.cnblogs.com/hicjiajia/archive/...