Skip to content

Collaboration #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 54 commits into
base: nightly
Choose a base branch
from
Open

Collaboration #10

wants to merge 54 commits into from

Conversation

spreeker
Copy link
Contributor

All merge conflicts resolved. Most likely still some errors left.

@Attumm Attumm changed the base branch from master to nightly April 13, 2021 08:38
- Easy to deploy.
- Easy to customize.
- Easy export data

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great to to read!!!

curlgeotest.sh Outdated
]
]
}' \
'http://127.0.0.1:8000/list/?groupby=postcode&reduce=count'
Copy link
Owner

@Attumm Attumm Apr 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this file is right now only relevant for one project it and not other projects, it can't be added.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well it s a first basic test that should be converted to a real test using real data. It took a long time to create like it is. Also serves as an example. so mark as NEEDS WORK or something.

curltest.sh Outdated
curl -vv 'http://127.0.0.1:8000/list/?groupby=woning_type&reduce=count'

# should not be cached.(using bitmaps)
curl -vv 'http://127.0.0.1:8000/list/?match-wijkcode=WK036394&groupby=woning_type&reduce=count'
Copy link
Owner

@Attumm Attumm Apr 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this file is right now only relevant for one project it and not other projects, it can't be added.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could add test dataset and convert them to proper tests.

}
result["woningenquivalent"] = strconv.Itoa(weq)
return result
}
Copy link
Owner

@Attumm Attumm Apr 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this file is right now only relevant for one project it and not other projects, it can't be added.

func (i Item) Row() []string {

lock.RLock()
defer lock.RUnlock()
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should try not to use locks. Mutexes will create bottlenecks for performance.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we use a translation from internal Items to ItemsOut while still reading or creating a race condition can occur then doing a translation from Item to ItemOut. Thats why this read lock is needed. It doesn't cause a notible delay IMHO. only used to output 'some' rows in the end for viewing. It is not executed millions of times.

}

func main() {
SETTINGS.Set("http_db_host", "0.0.0.0:8128", "host with port")
SETTINGS.Set("http_db_host", "0.0.0.0:8000", "host with port")
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default port of lambdadb is now 8128.
Because it's an perfect number.

https://en.wikipedia.org/wiki/8128_(number)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hahahahahah

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perfect it is.


//Construct yes or no to booleans in SETTINGS
SETTINGS.Set("readonly", "yes", "only allow read only funcions")
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

readonly is changed to mgmt.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

merge leftover :)


//Construct yes or no to booleans in SETTINGS
SETTINGS.Set("readonly", "yes", "only allow read only funcions")
SETTINGS.Set("debug", "no", "print memory usage")
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Debug is already used.


ITEMS = make(Items, 0, 100*1000)
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ITEMS should always exits in the heap.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I moved all Items code to store.go. Also inital size of Items leaves a lot of nil values when doing append. Since new items are added behind. if i rember correctly maybe some tests are needed :)

model.go Outdated
func sortBy(items Items, sortingL []string) (Items, []string) {

lock.Lock()
defer lock.Unlock()
Copy link
Owner

@Attumm Attumm Apr 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lock is not needed here.
Sorting should not be done on the heap ITEMS.

I don't know how to respond so im updating this comment, I can only press resolve?!

Let's create 1 large read write lock for the api's
block all requests while ingestion and indexes are created.
Reading from maps from multiple places is fine, only writing is not.

Since we only write data while ingestion and during the create, let's block all reading calls.
That way we can remove all these locks, locks are dangerous they can cause very painfull degradation of performance on production.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You sometimes need to sort on the actual values of an Item and not by the integer internal values of the Internal Item. Item.IntfieldtoRepteadedvalues -> ItemOut.actualvalue. There is a hashmap lookup needed. So similar as above when loading data and doing simultaneous request to see how it's going an race condition can occure where more goroutines are reading / writing to a hashmap.

BuurtcodeItems[i.Buurtcode] = ba
}
ba.SetBit(uint64(i.Label))
*/
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

woops, still a comment.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left that as documentation as to see what kind of code to expect there. Made it easier to write and debug model generating code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants