feat: allow adding indexes in configuration files

README.md

@@ -42,7 +42,7 @@ A model interchange project is a collection of SysML or KerML files with
 additional metadata such as project name, versions, and the list of projects on
 which it depends. To create a new project called `my_project` run:
 
-```bash
+```console
 $ sysand new my_project
     Creating interchange project `my_project`
 ```
@@ -52,7 +52,7 @@ interchange project, consisting of two files `.project.json` and `.meta.json`.
 
 Inside the directory, we can ask for a basic overview of the project.
 
-```bash
+```console
 $ cd my_project
 $ sysand info
 Name: my_project
@@ -65,7 +65,7 @@ No usages.
 The project we created in the previous subsection contains no source files as
 can be seen by running the following command:
 
-```bash
+```console
 $ sysand sources
 <NO OUTPUT>
 ```
@@ -80,15 +80,15 @@ package MyProject;
 Now, we can add `MyProject.sysml` to our project by running the following
 command:
 
-```bash
+```console
 $ sysand include MyProject.sysml
    Including files: ["MyProject.sysml"]
 ```
 
 The file will now be listed by `sysand sources`, which can serve as the input
 to a SysML v2 processing environment.
 
-```bash
+```console
 $ sysand sources
 /path/to/my_project/MyProject.sysml
 ```
@@ -110,10 +110,10 @@ install from the [Sysand Package Index](https://sysand.org). You can find the
 IRI (and the full install command) in the card of the package on the index
 website. For example, to install the standard Function Library, run:
 
-```bash
+```console
 $ sysand add urn:kpar:function-library
       Adding usage: urn:kpar:function-library
-     Creating env
+    Creating env
      Syncing env
   Installing urn:kpar:semantic-library 1.0.0
   Installing urn:kpar:data-type-library 1.0.0
@@ -123,7 +123,7 @@ $ sysand add urn:kpar:function-library
 It is also possible to install packages from the URL that points to the `.kpar`
 file as shown in the following snippet:
 
-```bash
+```console
 $ sysand add https://www.omg.org/spec/KerML/20250201/Function-Library.kpar
       Adding usage: https://www.omg.org/spec/KerML/20250201/Function-Library.kpar
     Creating env
@@ -135,19 +135,19 @@ $ sysand add https://www.omg.org/spec/KerML/20250201/Function-Library.kpar
 
 Adding a dependency may take a few seconds to run, as it will find and install
 the project (and any transitive usages) into a new local environment. Once
-finished, this will have created a file called `SysandLock.toml` and a directory
+finished, this will have created a file called `sysand-lock.toml` and a directory
 `sysand_env`. The former records the precise versions installed, so that the
 same installation can be reproduced later. The latter directory will contain a
 local installation of the added project, as well as any of its (transitive)
-usages. `SysandLock.toml` is sufficient to reproduce `sysand_env`; therefore, we
-recommend checking in `SysandLock.toml` into your version control system and
+usages. `sysand-lock.toml` is sufficient to reproduce `sysand_env`; therefore, we
+recommend checking in `sysand-lock.toml` into your version control system and
 adding `sysand_env` to `.gitignore`.
 
 We can confirm that the usage was successfully added by running the `info`
 command again:
 
-```bash
-> sysand info
+```console
+$ sysand info
 Name: my_project
 Version: 0.0.1
     Usage: https://www.omg.org/spec/KerML/20250201/Semantic-Library.kpar
@@ -156,7 +156,7 @@ Version: 0.0.1
 If we run `sysand source` again, it will now include all source files of the
 set of (transitive) dependencies.
 
-```bash
+```console
 $ sysand sources
 /Users/vakaras2/projects/tmp/sysand/sysand_env/7afe310696b522f251dc21ed6086ac4b50a663969fd1a49aa0aa2103d2a674ad/1.0.0.kpar/Metaobjects.kerml
 /Users/vakaras2/projects/tmp/sysand/sysand_env/7afe310696b522f251dc21ed6086ac4b50a663969fd1a49aa0aa2103d2a674ad/1.0.0.kpar/Performances.kerml
@@ -176,18 +176,18 @@ dependencies needed for a specific project.
 We can see everything installed in the local environment using `sysand env
 list`:
 
-```bash
+```console
 $ sysand env list
 https://www.omg.org/spec/KerML/20250201/Data-Type-Library.kpar 1.0.0
 https://www.omg.org/spec/KerML/20250201/Function-Library.kpar 1.0.0
 https://www.omg.org/spec/KerML/20250201/Semantic-Library.kpar 1.0.0
 ```
 
 If you want to recreate the environment on a new machine, make sure you have not
-only your project files, but also `SysandLock.toml` and execute the following
+only your project files, but also `sysand-lock.toml` and execute the following
 command:
 
-```bash
+```console
 $ sysand sync
     Creating env
      Syncing env
@@ -200,7 +200,7 @@ $ sysand sync
 
 To package your project for distribution, run `sysand build`:
 
-```bash
+```console
 $ sysand build
     Building kpar: /path/to/my_project/output/my_project.kpar
 ```
@@ -211,42 +211,48 @@ different project using `sysand`.
 ## Hosting a project index
 
 > [!important]
-> The structure of indices and `sysand_env` environments is still expected to
+> The structure of indexes and `sysand_env` environments is still expected to
 > change, and may currently not be compatible between sysand releases.
 
 The easiest way to host a project index from which to install packages is to
 expose a `sysand_env` over HTTP.
 
 If you have an existing `sysand_env`, and you have a working Python 3 environment
 you can test this with
-```
+
+```console
 $ python3 -m http.server -d sysand_env 8080
 Serving HTTP on 0.0.0.0 port 8080 (http://0.0.0.0:8080/) ...
 ```
 
 > [!important]
 > Python's built-in `http.server` module is *not* intended for production use.
 
-Any project in the above `sysand_env` can now be used in `sysand add`, `sysand sync`, 
-`sysand env install`, etc., as long as the flag `--use-index http://localhost:8080` is
-added (or soon by specifying it in `sysand.toml`!).
+Any project in the above `sysand_env` can now be used in `sysand add`, `sysand sync`,
+`sysand env install`, etc., as long as the flag `--use-index http://localhost:8080`
+is added (or soon by specifying it in `sysand.toml`!).
 
-For example, to create an index to publish the above `my_project` project we can create
-a fresh `sysand_env`.
-```
+For example, to create an index to publish the above `my_project` project we can
+create a fresh `sysand_env`.
+
+```console
 $ mkdir my_index
 $ cd my_index
 $ sysand env 
     Creating env
 ```
-Now we install `my_project`, specifying the IRI/URL that you want to use to refer to it:
-```
+
+Now we install `my_project`, specifying the IRI/URL that you want to use to refer
+to it:
+
+```console
 $ sysand env install urn:kpar:my_project --path /path/to/my_project/
   Installing urn:kpar:my_project 0.0.1
      Syncing env
 ```
-By default, this will also install any usages (dependencies) of `my_project`, you can use
-`--no-deps` to install only the project itself.
+
+By default, this will also install any usages (dependencies) of `my_project`, you
+can use `--no-deps` to install only the project itself.
 
 ## Documentation
 

core/src/config/mod.rs

@@ -2,6 +2,7 @@
 // SPDX-License-Identifier: MIT OR Apache-2.0
 
 use serde::{Deserialize, Serialize};
+use url::Url;
 
 #[cfg(feature = "filesystem")]
 pub mod local_fs;
@@ -19,6 +20,68 @@ impl Config {
         self.verbose = self.verbose.or(config.verbose);
         extend_option_vec(&mut self.index, config.index);
     }
+
+    pub fn index_urls(
+        &self,
+        index_urls: Vec<String>,
+        default_urls: Vec<String>,
+        default_override_urls: Vec<String>,
+    ) -> Result<Vec<Url>, url::ParseError> {
+        if default_override_urls.is_empty() {
+            self.index_urls_no_default_override(index_urls, default_urls)
+        } else {
+            self.index_urls_with_default_override(index_urls, default_override_urls)
+        }
+    }
+
+    fn index_urls_no_default_override(
+        &self,
+        index_urls: Vec<String>,
+        default_urls: Vec<String>,
+    ) -> Result<Vec<Url>, url::ParseError> {
+        let mut indexes: Vec<_> = self.index.iter().flat_map(|v| v.iter()).collect();
+
+        indexes.sort_by_key(|i| i.default.unwrap_or(false));
+
+        let has_default = indexes
+            .last()
+            .and_then(|index| index.default)
+            .unwrap_or(false);
+
+        let end: Vec<String> = if has_default {
+            std::iter::empty::<String>().collect()
+        } else {
+            default_urls
+        };
+
+        index_urls
+            .iter()
+            .map(|url| url.as_str())
+            .chain(indexes.iter().map(|i| i.url.as_str()))
+            .chain(end.iter().map(|url| url.as_str()))
+            .map(Url::parse)
+            .collect()
+    }
+
+    fn index_urls_with_default_override(
+        &self,
+        index_urls: Vec<String>,
+        default_urls: Vec<String>,
+    ) -> Result<Vec<Url>, url::ParseError> {
+        index_urls
+            .iter()
+            .map(|url| url.as_str())
+            .chain(
+                self.index
+                    .iter()
+                    .flat_map(|v| v.iter())
+                    .filter(|i| !i.default.unwrap_or(false))
+                    .map(|i| i.url.as_str()),
+            )
+            .chain(default_urls.iter().map(|url| url.as_str()))
+            .map(Url::parse)
+            .collect()
+    }
 }
 
 fn extend_option_vec<T>(target: &mut Option<Vec<T>>, src: Option<Vec<T>>) {
@@ -29,14 +92,16 @@ fn extend_option_vec<T>(target: &mut Option<Vec<T>>, src: Option<Vec<T>>) {
 
 #[derive(Debug, Default, Clone, PartialEq, Serialize, Deserialize)]
 pub struct Index {
-    name: Option<String>,
-    url: String,
-    explicit: Option<bool>,
-    default: Option<bool>,
+    pub name: Option<String>,
+    pub url: String,
+    // pub explicit: Option<bool>,
+    pub default: Option<bool>,
 }
 
 #[cfg(test)]
 mod tests {
+    use url::Url;
+
     use crate::config::{Config, Index};
 
     #[test]
@@ -54,7 +119,7 @@ mod tests {
 
         assert_eq!(index.name, None);
         assert_eq!(index.url, "");
-        assert_eq!(index.explicit, None);
+        // assert_eq!(index.explicit, None);
         assert_eq!(index.default, None);
     }
 
@@ -73,4 +138,99 @@ mod tests {
 
         assert_eq!(defaults, config);
     }
+
+    #[test]
+    fn index_urls_without_default() {
+        let config = Config {
+            index: Some(vec![Index {
+                url: "http://www.index.com".to_string(),
+                ..Default::default()
+            }]),
+            ..Default::default()
+        };
+        let index = vec!["http://www.extra-index.com".to_string()];
+        let default_urls = vec!["http://www.default.com".to_string()];
+        let default_override_urls = vec![];
+
+        let index_urls = config
+            .index_urls(index, default_urls, default_override_urls)
+            .unwrap();
+
+        assert_eq!(
+            index_urls,
+            vec![
+                Url::parse("http://www.extra-index.com").unwrap(),
+                Url::parse("http://www.index.com").unwrap(),
+                Url::parse("http://www.default.com").unwrap(),
+            ]
+        );
+    }
+
+    #[test]
+    fn index_urls_with_default() {
+        let config = Config {
+            index: Some(vec![
+                Index {
+                    url: "http://www.config-default.com".to_string(),
+                    default: Some(true),
+                    ..Default::default()
+                },
+                Index {
+                    url: "http://www.index.com".to_string(),
+                    ..Default::default()
+                },
+            ]),
+            ..Default::default()
+        };
+        let index = vec!["http://www.extra-index.com".to_string()];
+        let default_urls = vec!["http://www.default.com".to_string()];
+        let default_override_urls = vec![];
+
+        let index_urls = config
+            .index_urls(index, default_urls, default_override_urls)
+            .unwrap();
+
+        assert_eq!(
+            index_urls,
+            vec![
+                Url::parse("http://www.extra-index.com").unwrap(),
+                Url::parse("http://www.index.com").unwrap(),
+                Url::parse("http://www.config-default.com").unwrap(),
+            ]
+        );
+    }
+
+    #[test]
+    fn index_urls_with_override() {
+        let config = Config {
+            index: Some(vec![
+                Index {
+                    url: "http://www.config-default.com".to_string(),
+                    default: Some(true),
+                    ..Default::default()
+                },
+                Index {
+                    url: "http://www.index.com".to_string(),
+                    ..Default::default()
+                },
+            ]),
+            ..Default::default()
+        };
+        let index = vec!["http://www.extra-index.com".to_string()];
+        let default_urls = vec!["http://www.default.com".to_string()];
+        let default_override_urls = vec!["http://www.new-default.com".to_string()];
+
+        let index_urls = config
+            .index_urls(index, default_urls, default_override_urls)
+            .unwrap();
+
+        assert_eq!(
+            index_urls,
+            vec![
+                Url::parse("http://www.extra-index.com").unwrap(),
+                Url::parse("http://www.index.com").unwrap(),
+                Url::parse("http://www.new-default.com").unwrap(),
+            ]
+        );
+    }
 }