pypi package 'zxpy'

Popularity: Medium (more popular than 90% of all packages)
Description: Shell scripts made simple
Installation: pip install zxpy
Last version: 1.6.3 (Download)
Homepage: https://github.com/tusharsadhwani/zxpy
Size: 8.75 kB
License: MIT

Activity

Last modified: February 21, 2023 10:58 AM (7 months ago)
Versions released in one year: 1
Weekly downloads: 17
10/02/202201/15/202304/02/202306/18/202309/03/202302040608001234released versions / week
  • Versions released
  • Weekly downloads

What's new in version 1.6.3

Delta between version 1.6.2 and version 1.6.3

Source: Github
Commits:
  • 0d999adf9448415e179f47edf57c67f4d6f26b65, February 20, 2023 1:11 PM:
    Add requirements.txt (#51)
  • 5b0db6b7739b683ade721fa3f1ec555b1769eb03, February 21, 2023 7:53 AM:
    Make CLI arguments accessible in zxpy programs (#52)
    
    * Fix sys.argv passed to tests
    
    * Improve test assertion
    
    * Add $ arg support
    
    * Normalize /bin/sh as first argument of script_args
    
    * Add type def
  • ad22c8b748eb885527828387b0abf99a2d676fa9, February 21, 2023 10:55 AM:
    Add docs for shell args
  • 13adcc00f3ec15df16c4eb8540d033f21a90b974, February 21, 2023 10:55 AM:
    Bump version
Files changed:
README.md CHANGED
@@ -127,6 +127,43 @@ print(return_code) # 0
127
 
128
  More examples are in the [examples folder](./examples).
129
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
130
  ## Quoting
131
 
132
  Take this shell command:
127
 
128
  More examples are in the [examples folder](./examples).
129
 
130
+ ## CLI Arguments
131
+
132
+ When writing a shell script, you often want to pass CLI arguments to it.
133
+
134
+ Like so:
135
+
136
+ ```console
137
+ $ cat ./foo.sh
138
+ echo arg is: $1
139
+
140
+ $ ./foo.sh 123
141
+ arg is: 123
142
+ ```
143
+
144
+ To do the same in `zxpy`, pass the script arguments after a `--` in the `zxpy` CLI command.
145
+
146
+ ```python
147
+ #!/usr/bin/env zxpy
148
+
149
+ import sys
150
+ print("Argv is:", sys.argv)
151
+
152
+ ~"echo output: $1 $2 $3"
153
+ ```
154
+
155
+ ```console
156
+ $ ./test.py
157
+ Argv is: ['/bin/sh']
158
+ output:
159
+
160
+ $ ./test.py -- abc def
161
+ Argv is: ['/bin/sh', 'abc', 'def']
162
+ output: abc def
163
+ ```
164
+
165
+ Both `$1` and `sys.argv[1]` will do the same thing.
166
+
167
  ## Quoting
168
 
169
  Take this shell command:
requirements.txt CHANGED
@@ -0,0 +1 @@
 
1
+ .
setup.cfg CHANGED
@@ -1,6 +1,6 @@
1
  [metadata]
2
  name = zxpy
3
- version = 1.6.1
4
  description = Shell scripts made simple
5
  long_description = file: README.md
6
  long_description_content_type = text/markdown
1
  [metadata]
2
  name = zxpy
3
+ version = 1.6.3
4
  description = Shell scripts made simple
5
  long_description = file: README.md
6
  long_description_content_type = text/markdown
tests/test_files/argv.py CHANGED
@@ -1,4 +1,8 @@
1
  import sys
2
 
3
- assert len(sys.argv) == 1
4
- assert sys.argv[0].endswith("argv.py")
 
 
 
 
1
  import sys
2
 
3
+ assert len(sys.argv) == 3
4
+ assert sys.argv[1] == "foobar"
5
+ assert sys.argv[2] == "baz"
6
+
7
+ out = ~"echo $1 and $2"
8
+ assert out == "foobar and baz\n"
tests/test_files/injection.py CHANGED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
1
+ x = ~"uname -p"
2
+ print(x in ("arm\n", "x86_64\n"))
3
+
4
+ command = "uname -p"
5
+ _, _, rc = ~f"{command}" # This doesn't work
6
+ print(rc)
tests/zxpy_test.py CHANGED
@@ -79,7 +79,8 @@ def test_prints(capsys: pytest.CaptureFixture[str]) -> None:
79
 
80
  def test_argv() -> None:
81
  test_file = "./tests/test_files/argv.py"
82
- subprocess.run(["zxpy", test_file])
 
83
 
84
 
85
  def test_raise() -> None:
@@ -114,3 +115,60 @@ def f(n):
114
  assert stderr == b'\n'
115
  outlines = [line for line in stdout.decode().splitlines() if line.startswith('>>>')]
116
  assert outlines == [">>> hi", ">>> 10", ">>> ... ... ... ... >>> 8", ">>> "]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
79
 
80
  def test_argv() -> None:
81
  test_file = "./tests/test_files/argv.py"
82
+ returncode = subprocess.check_call(["zxpy", test_file, "--", "foobar", "baz"])
83
+ assert returncode == 0
84
 
85
 
86
  def test_raise() -> None:
115
  assert stderr == b'\n'
116
  outlines = [line for line in stdout.decode().splitlines() if line.startswith('>>>')]
117
  assert outlines == [">>> hi", ">>> 10", ">>> ... ... ... ... >>> 8", ">>> "]
118
+
119
+
120
+ @pytest.mark.parametrize(
121
+ ("input", "index", "output"),
122
+ (
123
+ ("echo 'hello world' hi", 0, False),
124
+ ("echo 'hello world' hi", 4, False),
125
+ ("echo 'hello world' hi", 5, True),
126
+ ("echo 'hello world' hi", 6, True),
127
+ ("echo 'hello world' hi", 16, True),
128
+ ("echo 'hello world' hi", 17, True),
129
+ ("echo 'hello world' hi", 18, False),
130
+ ("echo 'hello world' hi", 21, False),
131
+ ('abc "def\'ghi" jkl \'mnop\'', 5, False),
132
+ ('abc "def\'ghi" jkl \'mnop\'', 8, False),
133
+ ('abc "def\'ghi" jkl \'mnop\'', 10, False),
134
+ ('abc "def\'ghi" jkl \'mnop\'', 14, False),
135
+ ('abc "def\'ghi" jkl \'mnop\'', 17, False),
136
+ ('abc "def\'ghi" jkl \'mnop\'', 18, True),
137
+ ('abc "def\'ghi" jkl \'mnop\'', 21, True),
138
+ ("'a' 'b' c 'de' 'fg' h", 1, True),
139
+ ("'a' 'b' c 'de' 'fg' h", 3, False),
140
+ ("'a' 'b' c 'de' 'fg' h", 6, True),
141
+ ("'a' 'b' c 'de' 'fg' h", 10, False),
142
+ ("'a' 'b' c 'de' 'fg' h", 14, True),
143
+ ("'a' 'b' c 'de' 'fg' h", 16, False),
144
+ ("'a' 'b' c 'de' 'fg' h", 19, True),
145
+ ("'a' 'b' c 'de' 'fg' h", 22, False),
146
+ ("a \"b'c'd'e\" '\"' '\"abc'", 1, False),
147
+ ("a \"b'c'd'e\" '\"' '\"abc'", 2, False),
148
+ ("a \"b'c'd'e\" '\"' '\"abc'", 4, False),
149
+ ("a \"b'c'd'e\" '\"' '\"abc'", 6, False),
150
+ ("a \"b'c'd'e\" '\"' '\"abc'", 8, False),
151
+ ("a \"b'c'd'e\" '\"' '\"abc'", 10, False),
152
+ ("a \"b'c'd'e\" '\"' '\"abc'", 12, True),
153
+ ("a \"b'c'd'e\" '\"' '\"abc'", 13, True),
154
+ ("a \"b'c'd'e\" '\"' '\"abc'", 14, True),
155
+ ("a \"b'c'd'e\" '\"' '\"abc'", 15, False),
156
+ ("a \"b'c'd'e\" '\"' '\"abc'", 16, True),
157
+ ("a \"b'c'd'e\" '\"' '\"abc'", 17, True),
158
+ ("a \"b'c'd'e\" '\"' '\"abc'", 18, True),
159
+ ("a \"b'c'd'e\" '\"' '\"abc'", 20, True),
160
+ ),
161
+ )
162
+ def test_is_inside_single_quotes(input, index, output) -> None:
163
+ assert zx.is_inside_single_quotes(input, index) == output
164
+
165
+
166
+ def test_shell_injection():
167
+ """Test injecting commands or shell args like `$0` into shell strings."""
168
+ file = "./tests/test_files/injection.py"
169
+ output = subprocess.check_output(["zxpy", file, "--", "abc"]).decode()
170
+ assert output == (
171
+ "True\n" # uname -p worked as a string
172
+ "127\n" # uname -p inside f-string got quoted
173
+ )
174
+ # TODO: $1 injection test
zx.py CHANGED
@@ -29,6 +29,8 @@
29
  import codecs
30
  import contextlib
31
  import inspect
 
 
32
  import shlex
33
  import subprocess
34
  import sys
@@ -64,10 +66,21 @@ def cli() -> None:
64
  )
65
  parser.add_argument('filename', help='Name of file to run', nargs='?')
66
 
 
 
 
 
 
 
 
 
 
 
 
67
  args = parser.parse_args(namespace=ZxpyArgs())
68
 
69
- # Remove zxpy executable from argv
70
- del sys.argv[0]
71
 
72
  if args.filename is None:
73
  setup_zxpy_repl()
@@ -91,9 +104,70 @@ def cli() -> None:
91
  install()
92
 
93
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
94
  @contextlib.contextmanager
95
  def create_shell_process(command: str) -> Generator[IO[bytes], None, None]:
96
  """Creates a shell process, yielding its stdout to read data from."""
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
97
  process = subprocess.Popen(
98
  command,
99
  stdout=subprocess.PIPE,
29
  import codecs
30
  import contextlib
31
  import inspect
32
+ import pipes
33
+ import re
34
  import shlex
35
  import subprocess
36
  import sys
66
  )
67
  parser.add_argument('filename', help='Name of file to run', nargs='?')
68
 
69
+ # Everything passed after a `--` is arguments to be used by the script itself.
70
+ script_args = ['/bin/sh']
71
+ try:
72
+ separator_index = sys.argv.index('--')
73
+ script_args.extend(sys.argv[separator_index + 1 :])
74
+ # Remove everything after `--` so that argparse passes
75
+ sys.argv = sys.argv[:separator_index]
76
+ except ValueError:
77
+ # `--` not present in command, so no extra script args
78
+ pass
79
+
80
  args = parser.parse_args(namespace=ZxpyArgs())
81
 
82
+ # Once arg parsing is done, replace argv with script args
83
+ sys.argv = script_args
84
 
85
  if args.filename is None:
86
  setup_zxpy_repl()
104
  install()
105
 
106
 
107
+ def is_inside_single_quotes(string: str, index: int) -> bool:
108
+ """Returns True if the given index is inside single quotes in a shell command."""
109
+ quote_index = string.find("'")
110
+ if quote_index == -1:
111
+ # No single quotes
112
+ return False
113
+
114
+ if index < quote_index:
115
+ # We're before the start of the single quotes
116
+ return False
117
+
118
+ double_quote_index = string.find('"')
119
+ if double_quote_index >= 0 and double_quote_index < quote_index:
120
+ next_double_quote = string.find('"', double_quote_index + 1)
121
+ if next_double_quote == -1:
122
+ # Double quote opened but never closed
123
+ return False
124
+
125
+ # Single quotes didn't start and we passed the index
126
+ if next_double_quote >= index:
127
+ return False
128
+
129
+ # Ignore all single quotes inside double quotes.
130
+ index -= next_double_quote + 1
131
+ rest = string[next_double_quote + 1 :]
132
+ return is_inside_single_quotes(rest, index)
133
+
134
+ next_quote = string.find("'", quote_index + 1)
135
+ if next_quote >= index:
136
+ # We're inside single quotes
137
+ return True
138
+
139
+ index -= next_quote + 1
140
+ rest = string[next_quote + 1 :]
141
+ return is_inside_single_quotes(rest, index)
142
+
143
+
144
  @contextlib.contextmanager
145
  def create_shell_process(command: str) -> Generator[IO[bytes], None, None]:
146
  """Creates a shell process, yielding its stdout to read data from."""
147
+ # shell argument support, i.e. $0, $1 etc.
148
+
149
+ dollar_indices = [index for index, char in enumerate(command) if char == '$']
150
+ for dollar_index in reversed(dollar_indices):
151
+ if (
152
+ dollar_index >= 0
153
+ and dollar_index + 1 < len(command)
154
+ and command[dollar_index + 1].isdigit()
155
+ and not is_inside_single_quotes(command, dollar_index)
156
+ ):
157
+ end_index = dollar_index + 1
158
+ while end_index + 1 < len(command) and command[end_index + 1].isdigit():
159
+ end_index += 1
160
+
161
+ number = int(command[dollar_index + 1 : end_index + 1])
162
+
163
+ # Get argument number from sys.argv
164
+ if number < len(sys.argv):
165
+ replacement = sys.argv[number]
166
+ else:
167
+ replacement = ""
168
+
169
+ command = command[:dollar_index] + replacement + command[end_index + 1 :]
170
+
171
  process = subprocess.Popen(
172
  command,
173
  stdout=subprocess.PIPE,

Readme

zxpy

Shell scripts made simple 🐚

zxpy lets you seamlessly write shell commands inside Python code, to create readable and maintainable shell scripts.

Inspired by Google's zx, but made much simpler and more accessible using Python.

Rationale

Bash is cool, and it's extremely powerful when paired with linux coreutils and pipes. But apart from that, it's a whole another language to learn, and has a (comparatively) unintuitive syntax for things like conditionals and loops.

zxpy aims to supercharge bash by allowing you to write scripts in Python, but with native support for bash commands and pipes.

Let's use it to find all TODOs in one of my other projects, and format them into a table:

#! /usr/bin/env zxpy
todo_comments = ~"git grep -n TODO"
for todo in todo_comments.splitlines():
    filename, lineno, code = todo.split(':', 2)
    *_, comment = code.partition('TODO')
    print(f"{filename:40} on line {lineno:4}: {comment.lstrip(': ')}")

Running this, we get:

$ ./todo_check.py
README.md                                on line 154 : move this content somewhere more sensible.
instachat/lib/models/message.dart        on line 7   : rename to uuid
instachat/lib/models/update.dart         on line 13  : make int
instachat/lib/services/chat_service.dart on line 211 : error handling
server/api/api.go                        on line 94  : move these to /chat/@:address
server/api/user.go                       on line 80  : check for errors instead of relying on zero value

Writing something like this purely in bash or in Python would be much harder than this. Being able to use linux utilities seamlessly with a readable, general purpose language is what makes this a really powerful tool.

A larger, practical example

You can find a comparison between a practical-ish script written in bash and zxpy in EXAMPLE.md

Installation <a href="https://pypi.org/project/zxpy"><img src="https://img.shields.io/badge/pypi-zxpy-blue?style=flat"></a>

pip install zxpy

pipx

If you have pipx installed, you can try out zxpy without installing it, by running:

pipx run zxpy

Basic Examples

Make a file script.py (The name and extension can be anything):

#! /usr/bin/env zxpy
~'echo Hello world!'

file_count = ~'ls -1 | wc -l'
print("file count is:", file_count)

And then run it:

$ chmod +x ./script.py

$ ./script.py
Hello world!
file count is: 3

Run >>> help('zx') in Python REPL to find out more ways to use zxpy.

A slightly more involved example: run_all_tests.py

#! /usr/bin/env zxpy
test_files = (~"find -name '*_test\.py'").splitlines()

for filename in test_files:
    try:
        print(f'Running {filename:.<50}', end='')
        output = ~f'python {filename}'  # variables in your shell commands :D
        assert output == ''
        print('Test passed!')
    except:
        print(f'Test failed.')

Output:

$ ./run_all_tests.py
Running ./tests/python_version_test.py....................Test failed.
Running ./tests/platform_test.py..........................Test passed!
Running ./tests/imports_test.py...........................Test passed!

More examples are in EXAMPLE.md, and in the examples folder.

stderr and return codes

To get stderr and return code information out of the shell command, there is an alternative way of invoking the shell.

To use it, just use 3 variables on the left side of your ~'...' shell string:

stdout, stderr, return_code = ~'echo hi'
print(stdout)       # hi
print(return_code)  # 0

More examples are in the examples folder.

CLI Arguments

When writing a shell script, you often want to pass CLI arguments to it.

Like so:

$ cat ./foo.sh
echo arg is: $1

$ ./foo.sh 123
arg is: 123

To do the same in zxpy, pass the script arguments after a -- in the zxpy CLI command.

#!/usr/bin/env zxpy

import sys
print("Argv is:", sys.argv)

~"echo output: $1 $2 $3"
$ ./test.py
Argv is: ['/bin/sh']
output:

$ ./test.py -- abc def
Argv is: ['/bin/sh', 'abc', 'def']
output: abc def

Both $1 and sys.argv[1] will do the same thing.

Quoting

Take this shell command:

$ uname -a
Linux pop-os 5.11.0 [...] x86_64 GNU/Linux

Now take this piece of code:

>>> cmd = 'uname -a'
>>> ~f'{cmd}'
/bin/sh: 1: uname -a: not found

Why does this not work?

This is because uname -a was quoted into 'uname -a'. All values passed inside f-strings are automatically quoted to avoid shell injection.

To prevent quoting, the :raw format_spec can be used:

>>> cmd = 'uname -a'
>>> ~f'{cmd:raw}'
Linux pop-os 5.11.0 [...] x86_64 GNU/Linux

This disables quoting, and the command is run as-is as provided in the string.

Note that this shouldn't be used with external data, or this will expose you to shell injection.

Interactive mode

$ zxpy
zxpy shell
Python 3.8.5 (default, Jan 27 2021, 15:41:15)
[GCC 9.3.0]

>>> ~"ls | grep '\.py'"
__main__.py
setup.py
zx.py
>>>

Also works with path/to/python -m zx

It can also be used to start a zxpy session in an already running REPL. Simply do:

>>> import zx; zx.install()

and zxpy should be enabled in the existing session.

Development/Testing

To install from source, clone the repo, and do the following:

$ source ./venv/bin/activate  # Always use a virtualenv!
$ pip install -r requirements-dev.txt
Processing ./zxpy
[...]
Successfully installed zxpy-1.X.X
$ pytest  # runs tests