Add content to readmes and fix some markup

README.adoc

@@ -1,6 +1,6 @@
-= https://github.com/input-output-hk/marconi[Cardano Blockchain Indexer Framework]
+= https://github.com/input-output-hk/marconi[Marconi]
 :email: plutus@iohk.io
-:author: Input Output HK Limited
+:author: Input Output Global
 :toc: left
 :reproducible:
 
@@ -13,58 +13,65 @@ Its original purpose is to provide a library that eases the tailored indexing of
 information.
 However, this repository also contains a set of indexers that are ready to use
 and some executables that group them.
+We've also bundle some of them in executables that you can reuse if you want, as
+the noe we use for link:./marconi-sidechain/README.md[sidechain]
 While we'd recommend defining indexers that fits your need, you can reuse the
 existing ones to test Marconi.
 
 == Key features
 
 * Customizable backend. The core library comes with
-  file:./marconi-core/src/Marconi/Core/Indexer/SQLiteIndexer.hs[SQLite backend],
-  a file:./marconi-core/src/Marconi/Core/Indexer/FileIndexer.hs[file backend]
-  or a file:./marconi-core/src/Marconi/Core/Indexer/ListIndexer.hs[in-memory backend].
-* file:./marconi-core/src/Marconi/Core/Indexer/SQLiteAggregateQuery.hs[Group indexers]
+  link:./marconi-core/src/Marconi/Core/Indexer/SQLiteIndexer.hs[SQLite backend],
+  a link:./marconi-core/src/Marconi/Core/Indexer/FileIndexer.hs[file backend]
+  or a link:./marconi-core/src/Marconi/Core/Indexer/ListIndexer.hs[in-memory backend].
+* link:./marconi-core/src/Marconi/Core/Indexer/SQLiteAggregateQuery.hs[Group indexers]
   to query them,
-* file:./marconi-core/src/Marconi/Core/Preprocessor/Resume.hs[Resume indexing]
+* link:./marconi-core/src/Marconi/Core/Preprocessor/Resume.hs[Resume indexing]
   (new indexers don't reset existing ones).
-* file:./marconi-core/src/Marconi/Core/Transformer/WithCatchup.hs[Batch insert] to
+* link:./marconi-core/src/Marconi/Core/Transformer/WithCatchup.hs[Batch insert] to
   speed up synchronisation with the node.
 * Logging using
   https://github.com/input-output-hk/iohk-monitoring-framework[iohk-monitoring-framework].
-* Indexers can be file:./marconi-core/src/Marconi/Core/Coordinator.hs[grouped].
+* Indexers can be link:./marconi-core/src/Marconi/Core/Coordinator.hs[grouped].
+* Differentiate easily volatile data (subject to rollback) from stable one.
 
-A comparison with other chain-indexing projects is also https://example.com/TODO[available].
+A comparison with other chain-indexing projects is also
+link:./doc/read-the-docs-site/architecture/comparison-other-chain-indexing-projects.rst[available].
 
+////
 == Quick start
 
-To be completed...
+To be completed
+////
 
 == Documentation
 
 === User documentation
 
-The more in-depth user documentation is available in our http://example.com/TODO[website].
+A more complete user manual should be available soon.
 
 === Haddock documentation
 
 The Haskell API documentation (Haddock) for the Marconi libraries is hosted here:
 
 * https://input-output-hk.github.io/marconi/main/marconi-core[marconi-core]
+* https://input-output-hk.github.io/marconi/main/marconi-core-json-rpc[marconi-core-json-rpc]
 * https://input-output-hk.github.io/marconi/main/marconi-cardano-core[marconi-cardano-core]
 * https://input-output-hk.github.io/marconi/main/marconi-cardano-indexers[marconi-cardano-indexers]
-* https://input-output-hk.github.io/marconi/main/marconi-core-json-rpc[marconi-core-json-rpc]
 
 You may generate them directly with `Cabal` for each component:
 
 ```
 cabal haddock marconi-core
 cabal haddock marconi-core-json-rpc
-cabal haddock marconi-cardano-chain-index
+cabal haddock marconi-cardano-core
+cabal haddock marconi-cardano-indexers
 ```
 
 [[Architecture]]
 == Architecture overview
 
-See the http://example.com/TODO[Design and Architecture] section the website for more in-depth explainations.
+See the link:./doc/read-the-docs-site/architecture/index.rst[Design and Architecture] section the website for more in-depth explainations.
 
 == CONTRIBUTING
 
@@ -77,9 +84,11 @@ See link:CONTRIBUTING{outfilesuffix}[CONTRIBUTING] for comprehensive documentati
 
 Issues can be filed in the https://github.com/input-output-hk/marconi/issues[GitHub Issue tracker].
 
+////
 == Roadmap
 
 To be completed...
+////
 
 == Licensing
 

doc/read-the-docs-site/architecture/comparison-other-chain-indexing-projects.rst

@@ -10,20 +10,18 @@ Marconi provides few out-of-the-box indexing solutions and the existing
 executables were designed either as an example or to suit specific needs,
 not to address a general indexing need.
 
-cardano-node
-------------
+`cardano-node <https://github.com/input-output-hk/cardano-node>`_
+-----------------------------------------------------------------
 
-https://github.com/input-output-hk/cardano-node[``cardano-node``]
-doesn't offer any kind of indexing out-of-the-box.
+cardano-node doesn't offer any kind of indexing out-of-the-box.
 However, through the mini-protocols, one can get access to a stream of blocks
 that can be processed.
 The mini-protocol is what Marconi uses internally to synchronise with the node.
 
-cardano-db-sync
----------------
+`cardano-db-sync <https://github.com/input-output-hk/cardano-db-sync>`_
+-----------------------------------------------------------------------
 
-https://github.com/input-output-hk/cardano-db-sync[``cardano-db-sync``]
-is the reference indexing solution for Cardano.
+cardano-db-sync is the reference indexing solution for Cardano.
 It's exhaustive and is ready to use out of the box.
 Compared to Marconi, ``cardano-db-sync`` doesn't offer any way to customise the
 indexing (aside filtering out tables that you don't want to index) and
@@ -36,22 +34,21 @@ indepedently), while Marconi use a SQLite database embedded in the solution
 ``cardano-db-sync`` doesn't provide any kind of API on top of it, and many
 people uses ``blockfrost`` on top of it to access it through an API.
 
-Kupo
-----
+`Kupo <https://cardanosolutions.github.io/kupo/>`_
+--------------------------------------------------
 
-https://cardanosolutions.github.io/kupo/[Kupo] is also a predefined indexing solution.
+Kupo is also a predefined indexing solution.
 As Marconi, it uses ``SQLite`` as a backend.
 It also provides facilities to monitor addresses or policy ids
 for a short amount of time, but no further customisation.
 It doesn't have the modularity of Marconi, but it's an out of the box solutions
 that works.
 
 
-Scrolls
--------
+`Scrolls <https://github.com/txpipe/scrolls>`_
+----------------------------------------------
 
-https://github.com/txpipe/scrolls[Scrolls] is an aggregator of on-chain data,
-with a no-sql storage.
+Scrolls is an aggregator of on-chain data, with a no-sql storage.
 Use `node-to-node` protocol to synchronised, which is at the same time
 more defensive (thus slower) and more resilient than the node-to-client
 protocol of Marconi.
@@ -61,9 +58,9 @@ Scrolls is one step further than Marconi in the "one aggregator to
 answer one specific question" direction.
 
 
-Carp
-----
+`Carp <https://dcspark.github.io/carp/>`_
+-----------------------------------------
 
-https://dcspark.github.io/carp/[Carp] is another modular indexing solution.
+Carp is another modular indexing solution.
 It uses postgres database for storage and provides a set of indexers that you
 can choose from for indexing. They also provide a REST API to access the data.

marconi-cardano-core/README.md

@@ -0,0 +1,11 @@
+# marconi-cardano-core
+
+`marconi-cardano-core` is a Haskell library that ease the use of `marconi` on
+Cardano.
+
+It provides facilities to get the block streams from a node and a few helpers
+to connect indexers to this stream.
+
+It also contains a [test library](test-lib) that provides a mockchain that you
+can use to test your indexers.
+

marconi-cardano-indexers/README.md

@@ -0,0 +1,28 @@
+# marconi-cardano-indexers
+
+This package contains a set of indexers for Cardano that can be reused in any
+marconi-indexing project.
+
+## Available indexers
+
+- [`BlockInfo`](src/Marconi/Cardano/Indexers/BlockInfo.hs): get the starting
+  slot, the creation time and the epoch number of a block.
+- [`ChainTip`](src/Marconi/Cardano/Indexers/BlockInfo.hs) get the slot and block
+  header hash of the chain tip of the node your synchronising against.
+- [`Datum`](src/Marconi/Cardano/Indexers/Datum.hs): get all the datum on the
+  chain with their hash.
+- [`EpochNonce`](src/Marconi/Cardano/Indexers/EpochNonce.hs) get the epoch Nonce used
+  at each epoch.
+- [`EpochSDD`](src/Marconi/Cardano/Indexers/EpochSDD.hs) get the stake pool
+  delegation at each epoch.
+- [`MintTokenEvent`](src/Marconi/Cardano/Indexers/MintTokenEvent.hs) get all the
+  minting and/or burning events.
+- [`Spent`](src/Marconi/Cardano/Indexers/Spent.hs) get all the spent Tx outputs
+  with the transaction they were spent in.
+- [`Utxo`](src/Marconi/Cardano/Indexers/Spent.hs) get all the utxos that were
+  created.
+
+Note that both `EpochNonce` and `EpochSDD` require that you maintain a ledger
+state (that can be obtained using an
+[`ExtLedgerStateCoordinator`](src/Marconi/Cardano/Indexers/Spent.hs)), which is
+quite memory and CPU expensive, especially on mainnet.

marconi-cardano-indexers/src/Marconi/Cardano/Indexers/BlockInfo.hs

@@ -13,6 +13,7 @@ module Marconi.Cardano.Indexers.BlockInfo (
   BlockInfo (BlockInfo),
   blockNo,
   timestamp,
+  timestampUTCTime,
   epochNo,
 
   -- * Indexer and worker
@@ -49,6 +50,7 @@ import Data.Text (Text)
 import Data.Text qualified as Text
 import Data.Time qualified as Time
 import Data.Time.Clock.POSIX (POSIXTime)
+import Data.Time.Clock.POSIX qualified as Time
 import Data.Word (Word64)
 import Database.SQLite.Simple (NamedParam ((:=)))
 import Database.SQLite.Simple qualified as SQL
@@ -80,6 +82,9 @@ Aeson.deriveJSON Aeson.defaultOptions{Aeson.fieldLabelModifier = tail} ''BlockIn
 
 Lens.makeLenses ''BlockInfo
 
+timestampUTCTime :: Lens.Getter BlockInfo Time.UTCTime
+timestampUTCTime = timestamp . Lens.to (Time.posixSecondsToUTCTime . fromIntegral)
+
 type instance Core.Point BlockInfo = C.ChainPoint
 
 -- | A raw SQLite indexer for BlackInfo

marconi-core-json-rpc/README.md

@@ -1 +1,4 @@
 # marconi-core-json-rpc
+
+Provides types and function to ease the setup of JSON-RPC queries for an
+indexer.