-
Notifications
You must be signed in to change notification settings - Fork 4
Collaboration #10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: nightly
Are you sure you want to change the base?
Collaboration #10
Conversation
- Easy to deploy. | ||
- Easy to customize. | ||
- Easy export data | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great to to read!!!
curlgeotest.sh
Outdated
] | ||
] | ||
}' \ | ||
'http://127.0.0.1:8000/list/?groupby=postcode&reduce=count' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since this file is right now only relevant for one project it and not other projects, it can't be added.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well it s a first basic test that should be converted to a real test using real data. It took a long time to create like it is. Also serves as an example. so mark as NEEDS WORK or something.
curltest.sh
Outdated
curl -vv 'http://127.0.0.1:8000/list/?groupby=woning_type&reduce=count' | ||
|
||
# should not be cached.(using bitmaps) | ||
curl -vv 'http://127.0.0.1:8000/list/?match-wijkcode=WK036394&groupby=woning_type&reduce=count' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since this file is right now only relevant for one project it and not other projects, it can't be added.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could add test dataset and convert them to proper tests.
} | ||
result["woningenquivalent"] = strconv.Itoa(weq) | ||
return result | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since this file is right now only relevant for one project it and not other projects, it can't be added.
func (i Item) Row() []string { | ||
|
||
lock.RLock() | ||
defer lock.RUnlock() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we should try not to use locks. Mutexes will create bottlenecks for performance.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we use a translation from internal Items to ItemsOut while still reading or creating a race condition can occur then doing a translation from Item to ItemOut. Thats why this read lock is needed. It doesn't cause a notible delay IMHO. only used to output 'some' rows in the end for viewing. It is not executed millions of times.
} | ||
|
||
func main() { | ||
SETTINGS.Set("http_db_host", "0.0.0.0:8128", "host with port") | ||
SETTINGS.Set("http_db_host", "0.0.0.0:8000", "host with port") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The default port of lambdadb is now 8128.
Because it's an perfect number.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hahahahahah
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
perfect it is.
|
||
//Construct yes or no to booleans in SETTINGS | ||
SETTINGS.Set("readonly", "yes", "only allow read only funcions") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
readonly is changed to mgmt.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
merge leftover :)
|
||
//Construct yes or no to booleans in SETTINGS | ||
SETTINGS.Set("readonly", "yes", "only allow read only funcions") | ||
SETTINGS.Set("debug", "no", "print memory usage") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Debug is already used.
|
||
ITEMS = make(Items, 0, 100*1000) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ITEMS should always exits in the heap.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I moved all Items code to store.go. Also inital size of Items leaves a lot of nil values when doing append. Since new items are added behind. if i rember correctly maybe some tests are needed :)
model.go
Outdated
func sortBy(items Items, sortingL []string) (Items, []string) { | ||
|
||
lock.Lock() | ||
defer lock.Unlock() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lock is not needed here.
Sorting should not be done on the heap ITEMS.
I don't know how to respond so im updating this comment, I can only press resolve?!
Let's create 1 large read write lock for the api's
block all requests while ingestion and indexes are created.
Reading from maps from multiple places is fine, only writing is not.
Since we only write data while ingestion and during the create, let's block all reading calls.
That way we can remove all these locks, locks are dangerous they can cause very painfull degradation of performance on production.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You sometimes need to sort on the actual values of an Item and not by the integer internal values of the Internal Item. Item.IntfieldtoRepteadedvalues -> ItemOut.actualvalue. There is a hashmap lookup needed. So similar as above when loading data and doing simultaneous request to see how it's going an race condition can occure where more goroutines are reading / writing to a hashmap.
BuurtcodeItems[i.Buurtcode] = ba | ||
} | ||
ba.SetBit(uint64(i.Label)) | ||
*/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
woops, still a comment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I left that as documentation as to see what kind of code to expect there. Made it easier to write and debug model generating code.
…maps with more bitarray columns
All merge conflicts resolved. Most likely still some errors left.