Skip to content

Add support for Pluggable Discoveries #1333

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 74 commits into from
Aug 23, 2021
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
74 commits
Select commit Hold shift + click to select a range
386740b
[skip changelog] Add DiscoveryManager to PackageManager
silvanocerza Aug 9, 2021
ebca55f
Add loading of PluggableDiscoveries when loading a platform release
silvanocerza Jun 23, 2021
7cf3215
Added compatibility layer for non-pluggable platforms
cmaglie Jun 29, 2021
f442b1a
Implemented board list with discoveries
silvanocerza Aug 9, 2021
dd6a8fc
Implemented discovery loading after initialization
cmaglie Jun 29, 2021
f07775a
Implemented board watch with discoveries
silvanocerza Aug 9, 2021
e80869b
Fix load discoveries tests
silvanocerza Jun 29, 2021
23bab36
Fix some issues with board list watcher
silvanocerza Jun 30, 2021
3b4bd38
Fix FindToolsRequiredFromPlatformRelease not returning discoveries
silvanocerza Jun 30, 2021
47b84f5
Enhanced handling of some discoveries states
silvanocerza Jul 2, 2021
b32749f
Fix PackageManager reset
silvanocerza Jul 6, 2021
0d2add4
Add function to convert discovery.Port to rpc.Port
silvanocerza Aug 9, 2021
8bb84b2
Moved reference argument parsing to new package
silvanocerza Jul 6, 2021
3ff8287
Fix functions docstrings
silvanocerza Jul 8, 2021
ec3dfbc
Remove duplicated code to initialize Sketch path
silvanocerza Aug 9, 2021
fb934e5
Add property conversion for platform not supporting pluggable discovery
silvanocerza Jul 15, 2021
38c4325
Fix board list watch not working
silvanocerza Jul 15, 2021
a3007fe
Fix crash when converting Port to rpc struct
silvanocerza Jul 20, 2021
2d69063
Add generic Port argument
silvanocerza Jul 15, 2021
7e6dfca
Change gRPC upload functions to use new Port message
silvanocerza Aug 9, 2021
e6c33ea
Add support for upload user fields
silvanocerza Aug 9, 2021
4a586d9
Fix upload unit tests
silvanocerza Jul 20, 2021
67ae164
Fix code naming issues
silvanocerza Jul 20, 2021
ddf1603
Added builtin:mdns-discovery
cmaglie Jul 21, 2021
ea61dca
Do not panic if discovery tool is not installed
cmaglie Jul 21, 2021
d255c3d
Implemented port/protocol detection at CLI startup time
silvanocerza Aug 9, 2021
18777b2
Perform 1200bps-touch only on serial ports
cmaglie Jul 21, 2021
709db6f
Added missing properties for pluggable upload
cmaglie Jul 21, 2021
322ce23
Correctly implemented 'board list' timeout option
silvanocerza Aug 9, 2021
4fc7857
Updated mdns-discovery to 0.9.2
cmaglie Jul 22, 2021
77a40fd
Add documentation
silvanocerza Aug 10, 2021
f8ac21d
Add board properties to board list command and gRPC function
silvanocerza Aug 9, 2021
5ecf4d8
Fix documentation and code comments
silvanocerza Aug 10, 2021
c8fb6e4
Fix crash when attempting upload without specifying port address
silvanocerza Jul 23, 2021
2a7d3d2
Fix unit tests
silvanocerza Jul 23, 2021
015aea3
Update go-properties-orderedmap to fix discovery properties issues
silvanocerza Aug 11, 2021
33f8848
Fix more documentation
silvanocerza Jul 26, 2021
bd68d5e
Clarify pluggable discovery specification
silvanocerza Jul 26, 2021
69c6812
More documentation fixes
silvanocerza Jul 27, 2021
aec9695
Add upload_port properties docs in platform specification
silvanocerza Jul 27, 2021
ed318af
Change links from pluggable discovery RFC to official docs
silvanocerza Jul 27, 2021
95b69fa
Add more upload mock integration tests
silvanocerza Jul 27, 2021
3e3fcd2
Fix integration tests
silvanocerza Jul 28, 2021
b7d594f
Change property to declare pluggable discoveries
silvanocerza Aug 10, 2021
763ce6f
Change property to declare pluggable discoveries
silvanocerza Jul 28, 2021
ae3a497
Fix documentation
silvanocerza Jul 28, 2021
3f484ed
Fix loading of platform not supporting pluggable discovery
silvanocerza Jul 28, 2021
be9e879
Fix more documentation
silvanocerza Jul 29, 2021
225c123
Add pluggable discovery states documentation
silvanocerza Aug 2, 2021
569429d
Enhanced handling of pluggable discoveries states
silvanocerza Aug 9, 2021
a0f2aee
Discoveries processes are now killed if the HELLO command fails
silvanocerza Aug 3, 2021
2624785
Add pluggable discovery logging
silvanocerza Aug 4, 2021
2a555ff
Enhanced handling of failing pluggable discoveries
silvanocerza Aug 4, 2021
71dcc0e
Fix pluggable discoveries parallelization
silvanocerza Aug 9, 2021
a0a6371
Discoveries event channels are now created when start sync is called
silvanocerza Aug 9, 2021
cfc7aa4
Cached ports are now reset on discovery stop
silvanocerza Aug 9, 2021
2a34f7d
Renamed ListSync methods to ListCachedPorts
silvanocerza Aug 9, 2021
85607d6
Pluggable discovery upload user fields are now limited to 50 chars
silvanocerza Aug 9, 2021
a5be354
Fix i18n strings
silvanocerza Aug 9, 2021
e68e4e0
Fix failing integration tests
silvanocerza Aug 10, 2021
eef60aa
Fix i18n data
silvanocerza Aug 10, 2021
f8814bc
Fix integration tests again
silvanocerza Aug 10, 2021
5aa1f1d
[skip changelog] Internationalize strings added for pluggable discove…
per1234 Aug 10, 2021
9941402
Update docs/pluggable-discovery-specification.md
silvanocerza Aug 10, 2021
53e1d25
Fix failing workflows
silvanocerza Aug 11, 2021
9a355b7
Updated upload-mock tests for generation
cmaglie Aug 12, 2021
41db8b5
Added a lot of mock upload test (also with programmer option)
cmaglie Aug 16, 2021
b322874
test_upload_mock: Handle '{' and '}' in recipes
cmaglie Aug 16, 2021
d7a576c
network ota: autoconvert network_patter from legacy
cmaglie Aug 17, 2021
3c10109
Automatically add port detection properties for network discovery
cmaglie Aug 19, 2021
e6ee163
Slightly improved 'board list' text output
cmaglie Aug 20, 2021
75c7fef
Default 'board list' timeout to 1s
cmaglie Aug 20, 2021
c100e14
Added some code review fixes
cmaglie Aug 23, 2021
40943b1
Added unit test for legacy-package conversion to pluggable discovery
cmaglie Aug 23, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Added some code review fixes
  • Loading branch information
cmaglie committed Aug 23, 2021
commit c100e14d17e6b1f462ac0a82b0d0f3976f9f54d5
2 changes: 0 additions & 2 deletions arduino/cores/packagemanager/loader.go
Original file line number Diff line number Diff line change
Expand Up @@ -381,7 +381,6 @@ func convertLegacyPlatformToPluggableDiscovery(platform *cores.PlatformRelease)
oldUploadTool := board.Properties.Get("upload.tool")
if oldUploadTool == toolName && !board.Properties.ContainsKey("upload.tool.network") {
board.Properties.Set("upload.tool.network", convertedToolName)
// fmt.Printf("ADDED: %s.upload.tool.network=%s\n", board, convertedToolName)

// Add identification properties for network protocol
i := 0
Expand All @@ -392,7 +391,6 @@ func convertLegacyPlatformToPluggableDiscovery(platform *cores.PlatformRelease)
i++
}
board.Properties.Set(fmt.Sprintf("upload_port.%d.board", i), board.BoardID)
// fmt.Printf("ADDED: upload_port.%d.board=%s\n", i, board.BoardID)
}
}
}
Expand Down
34 changes: 19 additions & 15 deletions i18n/data/en.po
Original file line number Diff line number Diff line change
Expand Up @@ -369,7 +369,7 @@ msgstr "Config file written to: %s"
msgid "Configuring platform"
msgstr "Configuring platform"

#: cli/board/list.go:182
#: cli/board/list.go:183
msgid "Connected"
msgstr "Connected"

Expand Down Expand Up @@ -483,7 +483,7 @@ msgstr "Directory where to save generated files. Default is './docs', the direct
msgid "Disable completion description for shells that support it"
msgstr "Disable completion description for shells that support it"

#: cli/board/list.go:183
#: cli/board/list.go:184
msgid "Disconnected"
msgstr "Disconnected"

Expand Down Expand Up @@ -1447,6 +1447,10 @@ msgstr "Programmers:"
msgid "Progress {0}"
msgstr "Progress {0}"

#: cli/board/list.go:125
msgid "Protocol"
msgstr "Protocol"

#: cli/lib/search.go:174
msgid "Provides includes: %s"
msgstr "Provides includes: %s"
Expand Down Expand Up @@ -1812,7 +1816,7 @@ msgstr "Uninstalls one or more cores and corresponding tool dependencies if no l
msgid "Uninstalls one or more libraries."
msgstr "Uninstalls one or more libraries."

#: cli/board/list.go:156
#: cli/board/list.go:157
msgid "Unknown"
msgstr "Unknown"

Expand Down Expand Up @@ -2128,7 +2132,7 @@ msgstr "can't find latest release of %s"
msgid "can't find main Sketch file in %s"
msgstr "can't find main Sketch file in %s"

#: arduino/cores/packagemanager/loader.go:751
#: arduino/cores/packagemanager/loader.go:749
msgid "can't find pattern for discovery with id %s"
msgstr "can't find pattern for discovery with id %s"

Expand Down Expand Up @@ -2238,7 +2242,7 @@ msgstr "copying output file %s"
msgid "could not find a valid build artifact"
msgstr "could not find a valid build artifact"

#: arduino/cores/packagemanager/loader.go:723
#: arduino/cores/packagemanager/loader.go:721
msgid "creating discovery: %s"
msgstr "creating discovery: %s"

Expand Down Expand Up @@ -2293,11 +2297,11 @@ msgstr "destination dir %s already exists, cannot install"
msgid "discovery %[1]s process not started: %[2]w"
msgstr "discovery %[1]s process not started: %[2]w"

#: arduino/cores/packagemanager/loader.go:712
#: arduino/cores/packagemanager/loader.go:710
msgid "discovery not found: %s"
msgstr "discovery not found: %s"

#: arduino/cores/packagemanager/loader.go:717
#: arduino/cores/packagemanager/loader.go:715
msgid "discovery not installed: %s"
msgstr "discovery not installed: %s"

Expand Down Expand Up @@ -2496,7 +2500,7 @@ msgstr "getting build properties for board %[1]s: %[2]s"
msgid "getting discovery dependencies for platform %[1]s: %[2]s"
msgstr "getting discovery dependencies for platform %[1]s: %[2]s"

#: arduino/cores/packagemanager/loader.go:651
#: arduino/cores/packagemanager/loader.go:649
msgid "getting parent dir of %[1]s: %[2]s"
msgstr "getting parent dir of %[1]s: %[2]s"

Expand Down Expand Up @@ -2758,7 +2762,7 @@ msgstr "loading board data: %s"
msgid "loading boards: %s"
msgstr "loading boards: %s"

#: arduino/cores/packagemanager/loader.go:606
#: arduino/cores/packagemanager/loader.go:604
msgid "loading bundled tools from %[1]s: %[2]s"
msgstr "loading bundled tools from %[1]s: %[2]s"

Expand All @@ -2785,7 +2789,7 @@ msgstr "loading platform release %[1]s: %[2]s"
msgid "loading platform.txt: %v"
msgstr "loading platform.txt: %v"

#: arduino/cores/packagemanager/loader.go:573
#: arduino/cores/packagemanager/loader.go:571
msgid "loading tool release in %[1]s: %[2]s"
msgstr "loading tool release in %[1]s: %[2]s"

Expand Down Expand Up @@ -2997,7 +3001,7 @@ msgstr "platform not found: %s"

#: arduino/cores/packagemanager/install_uninstall.go:65
#: arduino/cores/packagemanager/install_uninstall.go:108
#: arduino/cores/packagemanager/loader.go:435
#: arduino/cores/packagemanager/loader.go:433
#: commands/compile/compile.go:128
msgid "platform not installed"
msgstr "platform not installed"
Expand Down Expand Up @@ -3050,7 +3054,7 @@ msgstr "quitting discovery %[1]s: %[2]w"
msgid "reading %[1]s directory: %[2]s"
msgstr "reading %[1]s directory: %[2]s"

#: arduino/cores/packagemanager/loader.go:656
#: arduino/cores/packagemanager/loader.go:654
msgid "reading %[1]s: %[2]s"
msgstr "reading %[1]s: %[2]s"

Expand All @@ -3064,7 +3068,7 @@ msgid "reading dir %[1]s: %[2]s"
msgstr "reading dir %[1]s: %[2]s"

#: arduino/cores/packagemanager/loader.go:162
#: arduino/cores/packagemanager/loader.go:564
#: arduino/cores/packagemanager/loader.go:562
msgid "reading directory %[1]s: %[2]s"
msgstr "reading directory %[1]s: %[2]s"

Expand Down Expand Up @@ -3163,7 +3167,7 @@ msgstr "saving downloaded index signature: %s"
msgid "scanning examples: %s"
msgstr "scanning examples: %s"

#: arduino/cores/packagemanager/loader.go:642
#: arduino/cores/packagemanager/loader.go:640
msgid "searching for builtin_tools_versions.txt in %[1]s: %[2]s"
msgstr "searching for builtin_tools_versions.txt in %[1]s: %[2]s"

Expand All @@ -3184,7 +3188,7 @@ msgstr "signature verification error: %s"
msgid "sketchPath"
msgstr "sketchPath"

#: arduino/cores/packagemanager/loader.go:498
#: arduino/cores/packagemanager/loader.go:496
msgid "skipping loading of boards %s: malformed custom board options"
msgstr "skipping loading of boards %s: malformed custom board options"

Expand Down
8 changes: 4 additions & 4 deletions i18n/rice-box.go

Large diffs are not rendered by default.

28 changes: 9 additions & 19 deletions test/test_upload_mock.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@
import sys
import hashlib
import pytest
import os
from pathlib import Path


Expand Down Expand Up @@ -462,24 +461,24 @@ def generate_build_dir(sketch_path):
"-f "
'"{data_dir}/packages/arduino/hardware/samd/1.8.11/variants/arduino_zero/openocd_scripts/arduino_zero.cfg" '
'-c "telnet_port disabled; program '
"{bo}{build_dir}/{sketch_name}.ino.bin{bc} verify reset "
'0x2000; shutdown"\n',
"{{{build_dir}/{sketch_name}.ino.bin}} verify reset 0x2000; "
'shutdown"\n',
"linux": '"{data_dir}/packages/arduino/tools/openocd/0.10.0-arduino7/bin/openocd" '
"-d2 -s "
'"{data_dir}/packages/arduino/tools/openocd/0.10.0-arduino7/share/openocd/scripts/" '
"-f "
'"{data_dir}/packages/arduino/hardware/samd/1.8.11/variants/arduino_zero/openocd_scripts/arduino_zero.cfg" '
'-c "telnet_port disabled; program '
"{bo}{build_dir}/{sketch_name}.ino.bin{bc} verify reset "
'0x2000; shutdown"\n',
"{{{build_dir}/{sketch_name}.ino.bin}} verify reset 0x2000; "
'shutdown"\n',
"win32": '"{data_dir}/packages/arduino/tools/openocd/0.10.0-arduino7/bin/openocd.exe" '
"-d2 -s "
'"{data_dir}/packages/arduino/tools/openocd/0.10.0-arduino7/share/openocd/scripts/" '
"-f "
'"{data_dir}/packages/arduino/hardware/samd/1.8.11/variants/arduino_zero/openocd_scripts/arduino_zero.cfg" '
'-c "telnet_port disabled; program '
"{bo}{build_dir}/{sketch_name}.ino.bin{bc} verify reset "
'0x2000; shutdown"\n',
"{{{build_dir}/{sketch_name}.ino.bin}} verify reset 0x2000; "
'shutdown"\n',
},
),
(
Expand Down Expand Up @@ -1170,7 +1169,7 @@ def test_upload_sketch(
}

# Install everything just once
if not os.path.isdir(session_data_dir + "/packages"):
if not Path(session_data_dir, "packages").is_dir():
assert run_command("config init --overwrite", custom_env=env)
for package_index in indexes:
assert run_command(f"config add board_manager.additional_urls {package_index}", custom_env=env)
Expand All @@ -1186,15 +1185,8 @@ def test_upload_sketch(

# Fake compilation, we just need the folder to exist
build_dir = generate_build_dir(sketch_path)
if programmer != "":
programmer_arg = "-P " + programmer
else:
programmer_arg = ""
if upload_port != "":
port_arg = "-p " + upload_port
else:
port_arg = ""

programmer_arg = f"-P {programmer}" if programmer else ""
port_arg = f"-p {upload_port}" if upload_port else ""
res = run_command(f'upload {port_arg} {programmer_arg} -b {fqbn} "{sketch_path}" --dry-run -v', custom_env=env)
assert res.ok

Expand All @@ -1207,8 +1199,6 @@ def test_upload_sketch(
upload_port=upload_port,
build_dir=build_dir,
sketch_name=sketch_name,
bo="{",
bc="}",
).replace("\\", "/")

expected_output in res.stdout.replace("\\", "/").replace("\r", "")