feat: batch_mode config option to unlock 3x multiple problem solving speed by reusing openebs and observe namespace (1mins speed up), plus helm pull policy (1mins speed up)

[zhangddjs]

No description provided.

[zhangddjs]

@microsoft-github-policy-service agree

[Jackson Clark (HacksonClark)]

I'm running into this issue when running a problem with cli.py:

== Fault Injection ==
Traceback (most recent call last):
File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 493, in _make_request
conn.request(
File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/urllib3/connection.py", line 494, in request
self.endheaders()
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1314, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1074, in _send_output
self.send(msg)
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1018, in send
self.connect()
File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/docker/transport/unixconn.py", line 26, in connect
sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 841, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/urllib3/util/retry.py", line 474, in increment
raise reraise(type(error), error, _stacktrace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/urllib3/util/util.py", line 38, in reraise
raise value.with_traceback(tb)
File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 493, in _make_request
conn.request(
File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/urllib3/connection.py", line 494, in request
self.endheaders()
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1314, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1074, in _send_output
self.send(msg)
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1018, in send
self.connect()
File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/docker/transport/unixconn.py", line 26, in connect
sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 223, in _retrieve_server_version
return self.version(api_version=False)["ApiVersion"]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/docker/api/daemon.py", line 181, in version
return self._result(self._get(url), json=True)
^^^^^^^^^^^^^^
File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/docker/utils/decorators.py", line 44, in inner
return f(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 246, in _get
return self.get(url, **self._set_request_timeout(kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/requests/sessions.py", line 602, in get
return self.request("GET", url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/requests/adapters.py", line 682, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/jackson/Desktop/MSR-AIOpsLab/cli.py", line 141, in <module>
asyncio.run(main())
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/runners.py", line 194, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/base_events.py", line 664, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/Users/jackson/Desktop/MSR-AIOpsLab/cli.py", line 135, in main
await agent.set_problem()
File "/Users/jackson/Desktop/MSR-AIOpsLab/cli.py", line 89, in set_problem
self.init_problem(problem_id.strip())
File "/Users/jackson/Desktop/MSR-AIOpsLab/cli.py", line 101, in init_problem
problem_desc, _, apis = self.orchestrator.init_problem(problem_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jackson/Desktop/MSR-AIOpsLab/aiopslab/orchestrator/orchestrator.py", line 77, in init_problem
prob.inject_fault()
File "/Users/jackson/Desktop/MSR-AIOpsLab/aiopslab/orchestrator/problems/k8s_target_port_misconfig/target_port.py", line 43, in inject_fault
injector = VirtualizationFaultInjector(namespace=self.namespace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jackson/Desktop/MSR-AIOpsLab/aiopslab/generators/fault/inject_virtual.py", line 22, in __init__
self.docker = Docker()
^^^^^^^^
File "/Users/jackson/Desktop/MSR-AIOpsLab/aiopslab/service/dock.py", line 12, in __init__
self.client = docker.from_env()
^^^^^^^^^^^^^^^^^
File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/docker/client.py", line 94, in from_env
return cls(
^^^^
File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/docker/client.py", line 45, in __init__
self.api = APIClient(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 207, in __init__
self._version = self._retrieve_server_version()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jackson/Library/Caches/pypoetry/virtualenvs/aiopslab-85Fme-83-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 230, in _retrieve_server_version
raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

Thought it seems potentially unrelated. Let me know if you see this as well!

[Jackson Clark (HacksonClark)]

See #64 (comment)

[zhangddjs]

See #64 (comment)

Thank you for sharing, may I confirm about the reproduce steps, is such like this :

# Step1: clone repo and checkout
git clone git@github.com:zhangddjs/AIOpsLab.git --recurse-submodules
cd AIOpsLab; git checkout zhangdd/feat/batch_mode
# Step2: settle config.yml file
cp -a aiopslab/config.example.yml aiopslab/config.yml; vim aiopslab/config.yml
# Step3: start cli.py
poetry run python cli.py
# Step4: start problem
aiopslab> start k8s_target_port-misconfig-detection-1

And may I know does main branch also have the mentioned issues in your environment?

[Jackson Clark (HacksonClark)]

I don't believe the issue is in main.

[zhangddjs]

ok, let me try to reproduce, from my side currently looks ok

[Jackson Clark (HacksonClark)]

zhangddjs what environment are you using? I'm using a kubernetes cluster deployed on cloudlab. Are you using a real Kubernetes cluster or an emulated one like KinD?