prod image build changes

[Bowrna]

closes : #21100
related : #21100

This helps to build prod image using Breeze2

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

Looks cool :)

[Bowrna]

@potiuk

airflow/scripts/ci/libraries/_build_images.sh

Line 542 in 408a7d6

if [[ !${INSTALL_AIRFLOW_VERSION}=~ ^[0-9\.]+((a|b|rc|alpha|beta|pre)[0-9]+)?$ ]];then

Can you give me a few example airflow version that matches this pattern?

[potiuk]

Can you give me a few example airflow version that matches this pattern?

• 2.0.1alpha1
• 2.2.2a10
• 2.3.0rc1
• 2.2.0.rc3
• 2.2.3pre0

[Bowrna]

• 2.2.3pre

@potiuk

• 2.0.1alpha1
• 2.3.0rc1
  Only these patterns of version matches for me

[potiuk]

Ah. right - we assume we always have number :)

[potiuk]

UPDATED

[Bowrna]

@potiuk Could you explain to me what's happening in this code part? If GITHUB_USERNAME is not specified then will it take apache as its value?

airflow/scripts/ci/libraries/_build_images.sh

Lines 336 to 339 in 408a7d6

	echo"${token}"| docker_v login \
	--username "${GITHUB_USERNAME:-apache}" \
	--password-stdin \
	"ghcr.io"||true

[potiuk]

Nice :)

[potiuk]

Just a small nit. :). If you work on tests - feel fre to add them but I am pretty much ready to merge it.

[potiuk]

Just a small nit. :). If you work on tests - feel fre to add them but I am pretty much ready to merge it.

[Bowrna]

@potiuk Could you explain to me how this side effect is put to use?

airflow/dev/breeze/tests/test_build_image.py

Lines 47 to 52 in 5eb6335

	check_cache_and_write_mock.side_effect=lambdacache_key, default_value: (
	cache_keyincached_values,
	cached_values[cache_key] ifcache_keyincached_valueselsedefault_value,
	)
	read_from_cache_mock.side_effect=lambdaparam_name: cached_values.get(param_name)
	build_parameters=get_image_build_params(parameters)

I could understand that you are mocking the return of those functions but I couldn't understand how it's further used in the code. If possible could you tell me or direct me to some relevant place to learn about it

[potiuk]

I could understand that you are mocking the return of those functions but I couldn't understand how it's further used in the code. If possible could you tell me or direct me to some relevant place to learn about it

Sure - this is the key - lambda (in this case) is really an anonymous function:

This

 read_from_cache_mock.side_effect = lambda param_name: cached_values.get(param_name) 

Is equivalent of this (just written in a shorter way):

 def read_from_cache_mocked_function(param_name):
return cached_values.get(param_name) read_from_cache_mock.side_effect = read_from_cache_mocked_function

The "side-effect" really points to the function that will be called when the "read_from_cache_mock" funciton will be actually called. This allows to implement some logic in your test "mocks" - they become a bit smarter this way. Usually (in simple cases) your mocks can be told to return specific value:

read_from_cache_mock.return_value = x

Also you can tell it to return different values if called multiple times by assigning an iterable to return value:

read_from_cache_mock.return_value = [return_when_called_first_time, return_when_called_second_time, return_when_called_third_time]

And side_effect is the "smartest" way - you just provide your own (usually very simple) implementation of a function that can return different return values - for example based on the parameters that are passed.

In the case above, The "read_from_cache_mock" and "check_cache_and_write_mock" values are really very simple implementation tha cache the values in memory and replace the more "file-based" cache.

[Bowrna]

I see @potiuk thank you. I could better understand the code.

[Bowrna]

@potiuk I have tried to add test for prod image similar to what you have done for CI image. Do you think I can further improve with more tests into prod image? Could you give me suggestion on this?

[Bowrna]

@potiuk Could you share your review about this PR? I made relevant changes wrt recent Dockerfile optimisation

[potiuk]

Rebasing to re-run all tests :)

[Bowrna]

@potiuk Fixing the Breeze2 tests failure

[potiuk]

Wooho! 🎉

[Bowrna]

closes : #21100
related : #21100

This helps to build prod image using Breeze2

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

Looks cool :)

[Bowrna]

@potiuk

airflow/scripts/ci/libraries/_build_images.sh

Line 542 in 408a7d6

if [[ !${INSTALL_AIRFLOW_VERSION}=~ ^[0-9\.]+((a|b|rc|alpha|beta|pre)[0-9]+)?$ ]];then

Can you give me a few example airflow version that matches this pattern?

[potiuk]

Can you give me a few example airflow version that matches this pattern?

• 2.0.1alpha1
• 2.2.2a10
• 2.3.0rc1
• 2.2.0.rc3
• 2.2.3pre0

[Bowrna]

• 2.2.3pre

@potiuk

• 2.0.1alpha1
• 2.3.0rc1
  Only these patterns of version matches for me

[potiuk]

Ah. right - we assume we always have number :)

[potiuk]

UPDATED

[Bowrna]

@potiuk Could you explain to me what's happening in this code part? If GITHUB_USERNAME is not specified then will it take apache as its value?

airflow/scripts/ci/libraries/_build_images.sh

Lines 336 to 339 in 408a7d6

	echo"${token}"| docker_v login \
	--username "${GITHUB_USERNAME:-apache}" \
	--password-stdin \
	"ghcr.io"||true

[potiuk]

Nice :)

[potiuk]

Just a small nit. :). If you work on tests - feel fre to add them but I am pretty much ready to merge it.

[potiuk]

Just a small nit. :). If you work on tests - feel fre to add them but I am pretty much ready to merge it.

[Bowrna]

@potiuk Could you explain to me how this side effect is put to use?

airflow/dev/breeze/tests/test_build_image.py

Lines 47 to 52 in 5eb6335

	check_cache_and_write_mock.side_effect=lambdacache_key, default_value: (
	cache_keyincached_values,
	cached_values[cache_key] ifcache_keyincached_valueselsedefault_value,
	)
	read_from_cache_mock.side_effect=lambdaparam_name: cached_values.get(param_name)
	build_parameters=get_image_build_params(parameters)

I could understand that you are mocking the return of those functions but I couldn't understand how it's further used in the code. If possible could you tell me or direct me to some relevant place to learn about it

[potiuk]

I could understand that you are mocking the return of those functions but I couldn't understand how it's further used in the code. If possible could you tell me or direct me to some relevant place to learn about it

Sure - this is the key - lambda (in this case) is really an anonymous function:

This

 read_from_cache_mock.side_effect = lambda param_name: cached_values.get(param_name) 

Is equivalent of this (just written in a shorter way):

 def read_from_cache_mocked_function(param_name):
return cached_values.get(param_name) read_from_cache_mock.side_effect = read_from_cache_mocked_function

The "side-effect" really points to the function that will be called when the "read_from_cache_mock" funciton will be actually called. This allows to implement some logic in your test "mocks" - they become a bit smarter this way. Usually (in simple cases) your mocks can be told to return specific value:

read_from_cache_mock.return_value = x

Also you can tell it to return different values if called multiple times by assigning an iterable to return value:

read_from_cache_mock.return_value = [return_when_called_first_time, return_when_called_second_time, return_when_called_third_time]

And side_effect is the "smartest" way - you just provide your own (usually very simple) implementation of a function that can return different return values - for example based on the parameters that are passed.

In the case above, The "read_from_cache_mock" and "check_cache_and_write_mock" values are really very simple implementation tha cache the values in memory and replace the more "file-based" cache.

[Bowrna]

I see @potiuk thank you. I could better understand the code.

[Bowrna]

@potiuk I have tried to add test for prod image similar to what you have done for CI image. Do you think I can further improve with more tests into prod image? Could you give me suggestion on this?

[Bowrna]

@potiuk Could you share your review about this PR? I made relevant changes wrt recent Dockerfile optimisation

[potiuk]

Rebasing to re-run all tests :)

[Bowrna]

@potiuk Fixing the Breeze2 tests failure

[potiuk]

Wooho! 🎉

[Bowrna]

closes : #21100
related : #21100

This helps to build prod image using Breeze2

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

Looks cool :)

[Bowrna]

@potiuk

airflow/scripts/ci/libraries/_build_images.sh

Line 542 in 408a7d6

if [[ !${INSTALL_AIRFLOW_VERSION}=~ ^[0-9\.]+((a|b|rc|alpha|beta|pre)[0-9]+)?$ ]];then

Can you give me a few example airflow version that matches this pattern?

[potiuk]

Can you give me a few example airflow version that matches this pattern?

• 2.0.1alpha1
• 2.2.2a10
• 2.3.0rc1
• 2.2.0.rc3
• 2.2.3pre0

[Bowrna]

• 2.2.3pre

@potiuk

• 2.0.1alpha1
• 2.3.0rc1
  Only these patterns of version matches for me

[potiuk]

Ah. right - we assume we always have number :)

[potiuk]

UPDATED

[Bowrna]

@potiuk Could you explain to me what's happening in this code part? If GITHUB_USERNAME is not specified then will it take apache as its value?

airflow/scripts/ci/libraries/_build_images.sh

Lines 336 to 339 in 408a7d6

	echo"${token}"| docker_v login \
	--username "${GITHUB_USERNAME:-apache}" \
	--password-stdin \
	"ghcr.io"||true

[potiuk]

Nice :)

[potiuk]

Just a small nit. :). If you work on tests - feel fre to add them but I am pretty much ready to merge it.

[potiuk]

Just a small nit. :). If you work on tests - feel fre to add them but I am pretty much ready to merge it.

[Bowrna]

@potiuk Could you explain to me how this side effect is put to use?

airflow/dev/breeze/tests/test_build_image.py

Lines 47 to 52 in 5eb6335

	check_cache_and_write_mock.side_effect=lambdacache_key, default_value: (
	cache_keyincached_values,
	cached_values[cache_key] ifcache_keyincached_valueselsedefault_value,
	)
	read_from_cache_mock.side_effect=lambdaparam_name: cached_values.get(param_name)
	build_parameters=get_image_build_params(parameters)

I could understand that you are mocking the return of those functions but I couldn't understand how it's further used in the code. If possible could you tell me or direct me to some relevant place to learn about it

[potiuk]

I could understand that you are mocking the return of those functions but I couldn't understand how it's further used in the code. If possible could you tell me or direct me to some relevant place to learn about it

Sure - this is the key - lambda (in this case) is really an anonymous function:

This

 read_from_cache_mock.side_effect = lambda param_name: cached_values.get(param_name) 

Is equivalent of this (just written in a shorter way):

 def read_from_cache_mocked_function(param_name):
return cached_values.get(param_name) read_from_cache_mock.side_effect = read_from_cache_mocked_function

The "side-effect" really points to the function that will be called when the "read_from_cache_mock" funciton will be actually called. This allows to implement some logic in your test "mocks" - they become a bit smarter this way. Usually (in simple cases) your mocks can be told to return specific value:

read_from_cache_mock.return_value = x

Also you can tell it to return different values if called multiple times by assigning an iterable to return value:

read_from_cache_mock.return_value = [return_when_called_first_time, return_when_called_second_time, return_when_called_third_time]

And side_effect is the "smartest" way - you just provide your own (usually very simple) implementation of a function that can return different return values - for example based on the parameters that are passed.

In the case above, The "read_from_cache_mock" and "check_cache_and_write_mock" values are really very simple implementation tha cache the values in memory and replace the more "file-based" cache.

[Bowrna]

I see @potiuk thank you. I could better understand the code.

[Bowrna]

@potiuk I have tried to add test for prod image similar to what you have done for CI image. Do you think I can further improve with more tests into prod image? Could you give me suggestion on this?

[Bowrna]

@potiuk Could you share your review about this PR? I made relevant changes wrt recent Dockerfile optimisation

[potiuk]

Rebasing to re-run all tests :)

[Bowrna]

@potiuk Fixing the Breeze2 tests failure

[potiuk]

Wooho! 🎉

[Bowrna]

closes : #21100
related : #21100

This helps to build prod image using Breeze2

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

Looks cool :)

[Bowrna]

@potiuk

airflow/scripts/ci/libraries/_build_images.sh

Line 542 in 408a7d6

if [[ !${INSTALL_AIRFLOW_VERSION}=~ ^[0-9\.]+((a|b|rc|alpha|beta|pre)[0-9]+)?$ ]];then

Can you give me a few example airflow version that matches this pattern?

[potiuk]

Can you give me a few example airflow version that matches this pattern?

• 2.0.1alpha1
• 2.2.2a10
• 2.3.0rc1
• 2.2.0.rc3
• 2.2.3pre0

[Bowrna]

• 2.2.3pre

@potiuk

• 2.0.1alpha1
• 2.3.0rc1
  Only these patterns of version matches for me

[potiuk]

Ah. right - we assume we always have number :)

[potiuk]

UPDATED

[Bowrna]

@potiuk Could you explain to me what's happening in this code part? If GITHUB_USERNAME is not specified then will it take apache as its value?

airflow/scripts/ci/libraries/_build_images.sh

Lines 336 to 339 in 408a7d6

	echo"${token}"| docker_v login \
	--username "${GITHUB_USERNAME:-apache}" \
	--password-stdin \
	"ghcr.io"||true

[potiuk]

Nice :)

[potiuk]

Just a small nit. :). If you work on tests - feel fre to add them but I am pretty much ready to merge it.

[potiuk]

Just a small nit. :). If you work on tests - feel fre to add them but I am pretty much ready to merge it.

[Bowrna]

@potiuk Could you explain to me how this side effect is put to use?

airflow/dev/breeze/tests/test_build_image.py

Lines 47 to 52 in 5eb6335

	check_cache_and_write_mock.side_effect=lambdacache_key, default_value: (
	cache_keyincached_values,
	cached_values[cache_key] ifcache_keyincached_valueselsedefault_value,
	)
	read_from_cache_mock.side_effect=lambdaparam_name: cached_values.get(param_name)
	build_parameters=get_image_build_params(parameters)

I could understand that you are mocking the return of those functions but I couldn't understand how it's further used in the code. If possible could you tell me or direct me to some relevant place to learn about it

[potiuk]

I could understand that you are mocking the return of those functions but I couldn't understand how it's further used in the code. If possible could you tell me or direct me to some relevant place to learn about it

Sure - this is the key - lambda (in this case) is really an anonymous function:

This

 read_from_cache_mock.side_effect = lambda param_name: cached_values.get(param_name) 

Is equivalent of this (just written in a shorter way):

 def read_from_cache_mocked_function(param_name):
return cached_values.get(param_name) read_from_cache_mock.side_effect = read_from_cache_mocked_function

The "side-effect" really points to the function that will be called when the "read_from_cache_mock" funciton will be actually called. This allows to implement some logic in your test "mocks" - they become a bit smarter this way. Usually (in simple cases) your mocks can be told to return specific value:

read_from_cache_mock.return_value = x

Also you can tell it to return different values if called multiple times by assigning an iterable to return value:

read_from_cache_mock.return_value = [return_when_called_first_time, return_when_called_second_time, return_when_called_third_time]

And side_effect is the "smartest" way - you just provide your own (usually very simple) implementation of a function that can return different return values - for example based on the parameters that are passed.

In the case above, The "read_from_cache_mock" and "check_cache_and_write_mock" values are really very simple implementation tha cache the values in memory and replace the more "file-based" cache.

[Bowrna]

I see @potiuk thank you. I could better understand the code.

[Bowrna]

@potiuk I have tried to add test for prod image similar to what you have done for CI image. Do you think I can further improve with more tests into prod image? Could you give me suggestion on this?

[Bowrna]

@potiuk Could you share your review about this PR? I made relevant changes wrt recent Dockerfile optimisation

[potiuk]

Rebasing to re-run all tests :)

[Bowrna]

@potiuk Fixing the Breeze2 tests failure

[potiuk]

Wooho! 🎉

[Bowrna]

closes : #21100
related : #21100

This helps to build prod image using Breeze2

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

Looks cool :)

[Bowrna]

@potiuk

airflow/scripts/ci/libraries/_build_images.sh

Line 542 in 408a7d6

if [[ !${INSTALL_AIRFLOW_VERSION}=~ ^[0-9\.]+((a|b|rc|alpha|beta|pre)[0-9]+)?$ ]];then

Can you give me a few example airflow version that matches this pattern?

[potiuk]

Can you give me a few example airflow version that matches this pattern?

• 2.0.1alpha1
• 2.2.2a10
• 2.3.0rc1
• 2.2.0.rc3
• 2.2.3pre0

[Bowrna]

• 2.2.3pre

@potiuk

• 2.0.1alpha1
• 2.3.0rc1
  Only these patterns of version matches for me

[potiuk]

Ah. right - we assume we always have number :)

[potiuk]

UPDATED

[Bowrna]

@potiuk Could you explain to me what's happening in this code part? If GITHUB_USERNAME is not specified then will it take apache as its value?

airflow/scripts/ci/libraries/_build_images.sh

Lines 336 to 339 in 408a7d6

	echo"${token}"| docker_v login \
	--username "${GITHUB_USERNAME:-apache}" \
	--password-stdin \
	"ghcr.io"||true

[potiuk]

Nice :)

[potiuk]

Just a small nit. :). If you work on tests - feel fre to add them but I am pretty much ready to merge it.

[potiuk]

Just a small nit. :). If you work on tests - feel fre to add them but I am pretty much ready to merge it.

[Bowrna]

@potiuk Could you explain to me how this side effect is put to use?

airflow/dev/breeze/tests/test_build_image.py

Lines 47 to 52 in 5eb6335

	check_cache_and_write_mock.side_effect=lambdacache_key, default_value: (
	cache_keyincached_values,
	cached_values[cache_key] ifcache_keyincached_valueselsedefault_value,
	)
	read_from_cache_mock.side_effect=lambdaparam_name: cached_values.get(param_name)
	build_parameters=get_image_build_params(parameters)

I could understand that you are mocking the return of those functions but I couldn't understand how it's further used in the code. If possible could you tell me or direct me to some relevant place to learn about it

[potiuk]

I could understand that you are mocking the return of those functions but I couldn't understand how it's further used in the code. If possible could you tell me or direct me to some relevant place to learn about it

Sure - this is the key - lambda (in this case) is really an anonymous function:

This

 read_from_cache_mock.side_effect = lambda param_name: cached_values.get(param_name) 

Is equivalent of this (just written in a shorter way):

 def read_from_cache_mocked_function(param_name):
return cached_values.get(param_name) read_from_cache_mock.side_effect = read_from_cache_mocked_function

The "side-effect" really points to the function that will be called when the "read_from_cache_mock" funciton will be actually called. This allows to implement some logic in your test "mocks" - they become a bit smarter this way. Usually (in simple cases) your mocks can be told to return specific value:

read_from_cache_mock.return_value = x

Also you can tell it to return different values if called multiple times by assigning an iterable to return value:

read_from_cache_mock.return_value = [return_when_called_first_time, return_when_called_second_time, return_when_called_third_time]

And side_effect is the "smartest" way - you just provide your own (usually very simple) implementation of a function that can return different return values - for example based on the parameters that are passed.

In the case above, The "read_from_cache_mock" and "check_cache_and_write_mock" values are really very simple implementation tha cache the values in memory and replace the more "file-based" cache.

[Bowrna]

I see @potiuk thank you. I could better understand the code.

[Bowrna]

@potiuk I have tried to add test for prod image similar to what you have done for CI image. Do you think I can further improve with more tests into prod image? Could you give me suggestion on this?

[Bowrna]

@potiuk Could you share your review about this PR? I made relevant changes wrt recent Dockerfile optimisation

[potiuk]

Rebasing to re-run all tests :)

[Bowrna]

@potiuk Fixing the Breeze2 tests failure

[potiuk]

Wooho! 🎉

[Bowrna]

closes : #21100
related : #21100

This helps to build prod image using Breeze2

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

Looks cool :)

[Bowrna]

@potiuk

airflow/scripts/ci/libraries/_build_images.sh

Line 542 in 408a7d6

if [[ !${INSTALL_AIRFLOW_VERSION}=~ ^[0-9\.]+((a|b|rc|alpha|beta|pre)[0-9]+)?$ ]];then

Can you give me a few example airflow version that matches this pattern?

[potiuk]

Can you give me a few example airflow version that matches this pattern?

• 2.0.1alpha1
• 2.2.2a10
• 2.3.0rc1
• 2.2.0.rc3
• 2.2.3pre0

[Bowrna]

• 2.2.3pre

@potiuk

• 2.0.1alpha1
• 2.3.0rc1
  Only these patterns of version matches for me

[potiuk]

Ah. right - we assume we always have number :)

[potiuk]

UPDATED

[Bowrna]

@potiuk Could you explain to me what's happening in this code part? If GITHUB_USERNAME is not specified then will it take apache as its value?

airflow/scripts/ci/libraries/_build_images.sh

Lines 336 to 339 in 408a7d6

	echo"${token}"| docker_v login \
	--username "${GITHUB_USERNAME:-apache}" \
	--password-stdin \
	"ghcr.io"||true

[potiuk]

Nice :)

[potiuk]

Just a small nit. :). If you work on tests - feel fre to add them but I am pretty much ready to merge it.

[potiuk]

Just a small nit. :). If you work on tests - feel fre to add them but I am pretty much ready to merge it.

[Bowrna]

@potiuk Could you explain to me how this side effect is put to use?

airflow/dev/breeze/tests/test_build_image.py

Lines 47 to 52 in 5eb6335

	check_cache_and_write_mock.side_effect=lambdacache_key, default_value: (
	cache_keyincached_values,
	cached_values[cache_key] ifcache_keyincached_valueselsedefault_value,
	)
	read_from_cache_mock.side_effect=lambdaparam_name: cached_values.get(param_name)
	build_parameters=get_image_build_params(parameters)

I could understand that you are mocking the return of those functions but I couldn't understand how it's further used in the code. If possible could you tell me or direct me to some relevant place to learn about it

[potiuk]

I could understand that you are mocking the return of those functions but I couldn't understand how it's further used in the code. If possible could you tell me or direct me to some relevant place to learn about it

Sure - this is the key - lambda (in this case) is really an anonymous function:

This

 read_from_cache_mock.side_effect = lambda param_name: cached_values.get(param_name) 

Is equivalent of this (just written in a shorter way):

 def read_from_cache_mocked_function(param_name):
return cached_values.get(param_name) read_from_cache_mock.side_effect = read_from_cache_mocked_function

The "side-effect" really points to the function that will be called when the "read_from_cache_mock" funciton will be actually called. This allows to implement some logic in your test "mocks" - they become a bit smarter this way. Usually (in simple cases) your mocks can be told to return specific value:

read_from_cache_mock.return_value = x

Also you can tell it to return different values if called multiple times by assigning an iterable to return value:

read_from_cache_mock.return_value = [return_when_called_first_time, return_when_called_second_time, return_when_called_third_time]

And side_effect is the "smartest" way - you just provide your own (usually very simple) implementation of a function that can return different return values - for example based on the parameters that are passed.

In the case above, The "read_from_cache_mock" and "check_cache_and_write_mock" values are really very simple implementation tha cache the values in memory and replace the more "file-based" cache.

[Bowrna]

I see @potiuk thank you. I could better understand the code.

[Bowrna]

@potiuk I have tried to add test for prod image similar to what you have done for CI image. Do you think I can further improve with more tests into prod image? Could you give me suggestion on this?

[Bowrna]

@potiuk Could you share your review about this PR? I made relevant changes wrt recent Dockerfile optimisation

[potiuk]

Rebasing to re-run all tests :)

[Bowrna]

@potiuk Fixing the Breeze2 tests failure

[potiuk]

Wooho! 🎉

[Bowrna]

closes : #21100
related : #21100

This helps to build prod image using Breeze2

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

Looks cool :)

[Bowrna]

@potiuk

airflow/scripts/ci/libraries/_build_images.sh

Line 542 in 408a7d6

if [[ !${INSTALL_AIRFLOW_VERSION}=~ ^[0-9\.]+((a|b|rc|alpha|beta|pre)[0-9]+)?$ ]];then

Can you give me a few example airflow version that matches this pattern?

[potiuk]

Can you give me a few example airflow version that matches this pattern?

• 2.0.1alpha1
• 2.2.2a10
• 2.3.0rc1
• 2.2.0.rc3
• 2.2.3pre0

[Bowrna]

• 2.2.3pre

@potiuk

• 2.0.1alpha1
• 2.3.0rc1
  Only these patterns of version matches for me

[potiuk]

Ah. right - we assume we always have number :)

[potiuk]

UPDATED

[Bowrna]

@potiuk Could you explain to me what's happening in this code part? If GITHUB_USERNAME is not specified then will it take apache as its value?

airflow/scripts/ci/libraries/_build_images.sh

Lines 336 to 339 in 408a7d6

	echo"${token}"| docker_v login \
	--username "${GITHUB_USERNAME:-apache}" \
	--password-stdin \
	"ghcr.io"||true

[potiuk]

Nice :)

[potiuk]

Just a small nit. :). If you work on tests - feel fre to add them but I am pretty much ready to merge it.

[potiuk]

Just a small nit. :). If you work on tests - feel fre to add them but I am pretty much ready to merge it.

[Bowrna]

@potiuk Could you explain to me how this side effect is put to use?

airflow/dev/breeze/tests/test_build_image.py

Lines 47 to 52 in 5eb6335

	check_cache_and_write_mock.side_effect=lambdacache_key, default_value: (
	cache_keyincached_values,
	cached_values[cache_key] ifcache_keyincached_valueselsedefault_value,
	)
	read_from_cache_mock.side_effect=lambdaparam_name: cached_values.get(param_name)
	build_parameters=get_image_build_params(parameters)

I could understand that you are mocking the return of those functions but I couldn't understand how it's further used in the code. If possible could you tell me or direct me to some relevant place to learn about it

[potiuk]

I could understand that you are mocking the return of those functions but I couldn't understand how it's further used in the code. If possible could you tell me or direct me to some relevant place to learn about it

Sure - this is the key - lambda (in this case) is really an anonymous function:

This

 read_from_cache_mock.side_effect = lambda param_name: cached_values.get(param_name) 

Is equivalent of this (just written in a shorter way):

 def read_from_cache_mocked_function(param_name):
return cached_values.get(param_name) read_from_cache_mock.side_effect = read_from_cache_mocked_function

The "side-effect" really points to the function that will be called when the "read_from_cache_mock" funciton will be actually called. This allows to implement some logic in your test "mocks" - they become a bit smarter this way. Usually (in simple cases) your mocks can be told to return specific value:

read_from_cache_mock.return_value = x

Also you can tell it to return different values if called multiple times by assigning an iterable to return value:

read_from_cache_mock.return_value = [return_when_called_first_time, return_when_called_second_time, return_when_called_third_time]

And side_effect is the "smartest" way - you just provide your own (usually very simple) implementation of a function that can return different return values - for example based on the parameters that are passed.

In the case above, The "read_from_cache_mock" and "check_cache_and_write_mock" values are really very simple implementation tha cache the values in memory and replace the more "file-based" cache.

[Bowrna]

I see @potiuk thank you. I could better understand the code.

[Bowrna]

@potiuk I have tried to add test for prod image similar to what you have done for CI image. Do you think I can further improve with more tests into prod image? Could you give me suggestion on this?

[Bowrna]

@potiuk Could you share your review about this PR? I made relevant changes wrt recent Dockerfile optimisation

[potiuk]

Rebasing to re-run all tests :)

[Bowrna]

@potiuk Fixing the Breeze2 tests failure

[potiuk]

Wooho! 🎉

[Bowrna]

closes : #21100
related : #21100

This helps to build prod image using Breeze2

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

Looks cool :)

[Bowrna]

@potiuk

airflow/scripts/ci/libraries/_build_images.sh

Line 542 in 408a7d6

if [[ !${INSTALL_AIRFLOW_VERSION}=~ ^[0-9\.]+((a|b|rc|alpha|beta|pre)[0-9]+)?$ ]];then

Can you give me a few example airflow version that matches this pattern?

[potiuk]

Can you give me a few example airflow version that matches this pattern?

• 2.0.1alpha1
• 2.2.2a10
• 2.3.0rc1
• 2.2.0.rc3
• 2.2.3pre0

[Bowrna]

• 2.2.3pre

@potiuk

• 2.0.1alpha1
• 2.3.0rc1
  Only these patterns of version matches for me

[potiuk]

Ah. right - we assume we always have number :)

[potiuk]

UPDATED

[Bowrna]

@potiuk Could you explain to me what's happening in this code part? If GITHUB_USERNAME is not specified then will it take apache as its value?

airflow/scripts/ci/libraries/_build_images.sh

Lines 336 to 339 in 408a7d6

	echo"${token}"| docker_v login \
	--username "${GITHUB_USERNAME:-apache}" \
	--password-stdin \
	"ghcr.io"||true

[potiuk]

Nice :)

[potiuk]

Just a small nit. :). If you work on tests - feel fre to add them but I am pretty much ready to merge it.

[potiuk]

Just a small nit. :). If you work on tests - feel fre to add them but I am pretty much ready to merge it.

[Bowrna]

@potiuk Could you explain to me how this side effect is put to use?

airflow/dev/breeze/tests/test_build_image.py

Lines 47 to 52 in 5eb6335

	check_cache_and_write_mock.side_effect=lambdacache_key, default_value: (
	cache_keyincached_values,
	cached_values[cache_key] ifcache_keyincached_valueselsedefault_value,
	)
	read_from_cache_mock.side_effect=lambdaparam_name: cached_values.get(param_name)
	build_parameters=get_image_build_params(parameters)

I could understand that you are mocking the return of those functions but I couldn't understand how it's further used in the code. If possible could you tell me or direct me to some relevant place to learn about it

[potiuk]

I could understand that you are mocking the return of those functions but I couldn't understand how it's further used in the code. If possible could you tell me or direct me to some relevant place to learn about it

Sure - this is the key - lambda (in this case) is really an anonymous function:

This

 read_from_cache_mock.side_effect = lambda param_name: cached_values.get(param_name) 

Is equivalent of this (just written in a shorter way):

 def read_from_cache_mocked_function(param_name):
return cached_values.get(param_name) read_from_cache_mock.side_effect = read_from_cache_mocked_function

The "side-effect" really points to the function that will be called when the "read_from_cache_mock" funciton will be actually called. This allows to implement some logic in your test "mocks" - they become a bit smarter this way. Usually (in simple cases) your mocks can be told to return specific value:

read_from_cache_mock.return_value = x

Also you can tell it to return different values if called multiple times by assigning an iterable to return value:

read_from_cache_mock.return_value = [return_when_called_first_time, return_when_called_second_time, return_when_called_third_time]

And side_effect is the "smartest" way - you just provide your own (usually very simple) implementation of a function that can return different return values - for example based on the parameters that are passed.

In the case above, The "read_from_cache_mock" and "check_cache_and_write_mock" values are really very simple implementation tha cache the values in memory and replace the more "file-based" cache.

[Bowrna]

I see @potiuk thank you. I could better understand the code.

[Bowrna]

@potiuk I have tried to add test for prod image similar to what you have done for CI image. Do you think I can further improve with more tests into prod image? Could you give me suggestion on this?

[Bowrna]

@potiuk Could you share your review about this PR? I made relevant changes wrt recent Dockerfile optimisation

[potiuk]

Rebasing to re-run all tests :)

[Bowrna]

@potiuk Fixing the Breeze2 tests failure

[potiuk]

Wooho! 🎉