evals
AltStore
evals | AltStore | |
---|---|---|
49 | 823 | |
13,920 | 11,021 | |
2.5% | 4.1% | |
9.3 | 9.4 | |
11 days ago | 11 days ago | |
Python | Swift | |
GNU General Public License v3.0 or later | GNU Affero General Public License v3.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
evals
-
Show HN: Times faster LLM evaluation with Bayesian optimization
Fair question.
Evaluate refers to the phase after training to check if the training is good.
Usually the flow goes training -> evaluation -> deployment (what you called inference). This project is aimed for evaluation. Evaluation can be slow (might even be slower than training if you're finetuning on a small domain specific subset)!
So there are [quite](https://github.com/microsoft/promptbench) [a](https://github.com/confident-ai/deepeval) [few](https://github.com/openai/evals) [frameworks](https://github.com/EleutherAI/lm-evaluation-harness) working on evaluation, however, all of them are quite slow, because LLM are slow if you don't have infinite money. [This](https://github.com/open-compass/opencompass) one tries to speed up by parallelizing on multiple computers, but none of them takes advantage of the fact that many evaluation queries might be similar and all try to evaluate on all given queries. And that's where this project might come in handy.
- I asked 60 LLMs a set of 20 questions
-
Ask HN: How are you improving your use of LLMs in production?
OpenAI open sourced their evals framework. You can use it to evaluate different models but also your entire prompt chain setup. https://github.com/openai/evals
They also have a registry of evals built in.
-
SuperAlignment
"What if" is all these "existential risk" conversations ever are.
Where is your evidence that we're approaching human level AGI, let alone SuperIntelligence? Because ChatGPT can (sometimes) approximate sophisticated conversation and deep knowledge?
How about some evidence that ChatGPT isn't even close? Just clone and run OpenAI's own evals repo https://github.com/openai/evals on the GPT-4 API.
It performs terribly on novel logic puzzles and exercises that a clever child could learn to do in an afternoon (there are some good chess evals, and I submitted one asking it to simulate a Forth machine).
-
What is that new "Alpha" tab in ChatGPT Plus? Are limits gone for standard GPT-4???
Ah well, I think you just got lucky then, I did the same with the survey. I'll be compulsively checking mine all day today lol. People on Reddit like to say that if you did an Eval which is basically a performance test natively run using code on GPT models, then OpenAI is more likely to favor you when they’re releasing new features. If ydk, then I guess that answers that.
-
OpenAI Function calling and API updates
You can get GPT 4 access by submitting an eval if gets merged (https://github.com/openai/evals). Here's the one that got me access[1]
Although from the blog post it looks like they're planning to open up to everyone soon, so that may happen before you get through the evals backlog.
1: https://github.com/openai/evals/pull/778
- GitHub - openai/evals: Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
- There have been a lot of threads and comments around the models in ChatGPT and the API outputs getting much worse in the last few weeks. This is a huge reason why we open sourced https://github.com/openai/evals . You can write an eval and test the quality over time. No guesswork!
-
Spend time on openai evals - Community - OpenAI Developer Forum
来源:GitHub - openai/evals: Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks. 8
- Is it worth it to critique the dialogue chatgpt4 generates? I’m hoping the feedback I provide can somehow help it in future models. …Waste of time?
AltStore
- Apple must open iPadOS to sideloading within 6 months, EU says
-
A first look at Europe's alternative iPhone app stores
AltServer from the AltStore folks allows you to automate the renewal of sideloaded apps. It’s not perfect, but it’s an excellent workaround.
http://altstore.io/
-
More options for apps distributed in the European Union
That's probably to prevent the most obvious workaround of creating a new shell company for every million users. (Which would be not so ridiculous as it sounds, there is plenty of software you cannot buy directly but only through a reseller. Epic could become a pure b2b shop on paper and sell Fortnite clients to regional distributors, or something like that.)
Some time ago somebody made an alternative App Store for emulators, https://altstore.io . I think it works by having users get a developer's certificate and installing the apps like an in-development app. I think it would be really neat if this model got tested in court and declared completely legal.
-
Ask HN: Is it possible to build React Native for iOS without a Mac?
See: https://altstore.io/
2 apps, re-sign weekly maximum. 3 if you do it without AltStore. Unlimited with a $99/yr developer account.
-
Apple Announces Changes to iOS, Safari, and the App Store in the European Union
you already could in a way... by using altstore[0] on non jailbroken devices... It's not as straightforward as on Android but it is possible (there were even some builds of blink engine)
[0] https://github.com/altstoreio/AltStore
-
Do I need to get out the soldering-iron again? (2018)
I mean, that's fine. My argument about adblockers applies to other software too: the Apple ecosystem has some of the basics (like sync, browsers, etc) figured out for me so that I don't need to fiddle with it. While I'd like to use Firefox, I don't need to, and the tradeoffs that come with accepting Safari instead are worth it for my specific situation. Forcing myself into a different ecosystem so that I can use different software that does the same thing isn't a good tradeoff for me. It sounds like that's not the case for you - glad you've found an ecosystem that works for you.
There are a couple of things you might want to be aware of though:
* AltStore exists and works pretty well: https://altstore.io
* iOS 17.2 allows users in some locales to side load apps: https://medium.com/@rmndrathna4/ios-17-2-sideload-apps-what-... . This was sparked by the Digital Markets Act, which could also force Apple to allow alternate browser engines. It went into effect May 2023, but I'm not a lawyer and idk how this will actually affect the Apple ecosystem. https://en.wikipedia.org/wiki/Digital_Markets_Act
-
[Question] iPhone 4S Downgrade
No problem! I can guide you step by step here. You will need: A Mac Computer A supported Device (which I know you have) A jail broken device This tutorial is for iOS 9 btw, so you should probably update. It’ll only be temporary. First, jailbreak your iPhone. Personally, I updated my iPhone to iOS 9 then jail broke it using Phoenix. To do this, turn on your Mac and download Altstore. (https://altstore.io). Go to the phoenix website and download the ipa file. Then go into Altstore and sideload the ipa on your device by plugging in the 4S and trusting the Mac. It’ll ask you for your Apple ID and password, but if you don’t feel safe giving it, you can create a throw-away Apple ID. Once the ipa is done sideloading, you should see the Phoenix app icon on your main iPhone menu. Go into it, and it’ll say iPhone 4,1 isn’t jailbrolen. Click on the begin button, and go through the terms and service. They will show you their mixtape, but you can ignore it. Now, click begin installation. You should see two buttons to use the premade files or your own. Here, you MUST WAIT at LEAST 5 minutes on this screen before you proceed. If you click the button before five minutes, the screen would fade black and you did it incorrectly. After five minutes, click the use premade asserts button (the button above). After a bit, you should get a storage full message. Then the phone will shut off on its own. When the phone is on, you should have Cydia. Go back into the Phoenix app and make sure it says your iPhones is jailbroken and Cydia can be launched. If not, click the “jumpstart jailbreak” button, and when you get to the assets thing again, wait 5 minutes once more. Once that is done, go into Cydia and go to the sources button. Click Edit on the upper right hand corner, then click Add. Then type this link in. http://repo.tihmstar.net/. Once done, scroll through his selection until you find the kDFU app. Download it, and go into the app. Turn on all the switches, and enter kDFU mode. Now we go back to your Mac. Go here: https://github.com/LukeZGD/Legacy-iOS-Kit/releases/tag/latest. Click on the one that says _Macos. That should download the zip file. Now unzip it, but don’t go into it. Now go to the Mac Terminal. Type cd then a space. Then drag the iOS Kit folder to terminal. The path name should show in terminal next to cd. Click enter. Now type chmod +x restore.sh Now type ./restore.sh You should see some things pop up. Now read if it asks you to update. Type “y” if it asks, and let it do its thing. When it is done updating, type ./restore.sh again. Now if you iPhone 4S is still in kDFU mode and is plugged in, the Patcher should see your iPhone 4S. Now it should show you some options. Type “1” as you are downgrading and click enter. It should now show you if you wanna downgrade to iOS 8 or 6. Type “2” as that is the option listed for iOS 6. Click enter. Now you should see a list of options that show Ipsw related things. Type “2” as that should download the iOS 6 ipsw. When that is done, type “3” and click enter. That will begin the restore. It’ll ask you a few questions, and type y/n to what you want. Now let it finish. Do NOT unplug the phone. When it is done, you should be downgraded!
-
alternative to iOS Beta app?
/u/sevenlayercookie5 /u/rogo725 - I side load it with https://altstore.io/ - it keeps it refreshed every 7 days as well. If you need any help with it, hit me up.
-
IOS Emulator
https://altstore.io/ Go here in your computer, download alt server then connect your phone via usb and download AltStore on phone. AltStore will have delta available to download.
-
[Tutorial] How to setup AltServer on Raspberry Pi/Linux Box and sync your device wirelessly (2023)
Fuck you for not having a Linux version JKJK Thanks for Altserver and Altstore: https://github.com/altstoreio/AltStore
What are some alternatives?
gpt4-pdf-chatbot-langchain - GPT4 & LangChain Chatbot for large PDF docs
xManager-Spotify - Ad-Free, New Features & Freedom [Moved to: https://github.com/xManager-App/xManager]
promptfoo - Test your prompts, models, and RAGs. Catch regressions and improve prompt quality. LLM evals for OpenAI, Azure, Anthropic, Gemini, Mistral, Llama, Bedrock, Ollama, and other local & private models with CI/CD integration.
TrollStore - Jailed iOS app that can install IPAs permanently with arbitary entitlements and root helpers because it trolls Apple
RWKV-LM - RWKV is an RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). So it's combining the best of RNN and transformer - great performance, fast inference, saves VRAM, fast training, "infinite" ctx_len, and free sentence embedding.
SideStore - SideStore is a fork of AltStore that doesn't require an AltServer.
gpt4free - The official gpt4free repository | various collection of powerful language models
AltServer-Linux - AltServer for AltStore, but on-device
clownfish - Constrained Decoding for LLMs against JSON Schema
Satella - Modern in-app purchase cracker (iOS 12-16)
BIG-bench - Beyond the Imitation Game collaborative benchmark for measuring and extrapolating the capabilities of language models
uYouPlus - uYou+ is a modified version of uYou (made by @MiRO92) with additional features and mainly made for non jailbroken users!