dafny VS Pandas

Automation of Hoare logic is quite good these days. Dafny, from MS Research (https://dafny.org), is probably the most friendly formal language. Dafny has been used to verify large systems, including many components of AWS. I am hoping that LLMs make more advanced languages, such as Liquid Haskell or Agda, much easier to write. Ideally, lots of code should be autocompleted once a human provides a type signature. The advantage of formal verification is that we are sure the generated code is correct.

Modern software verification employs various approaches, each offering different trade-offs between ease of use and strength of guarantees. AWS contributes to the open source program verification tools used in the previous examples. Dafny and Kani represent two powerful approaches to program verification. Let's see how they work in practice before connecting the dots between automated reasoning and generative AI.

On that note, I discovered Dafny[1] recently, as a more accessible way to program with proofs. There's a companion book[2] that seems very accessible and down-to-earth (and, unfortunately, quite expensive). I didn't have the time to play with it yet, but it looks like it does what Ada/SPARK does (and more), but with less verbose syntax and more options for compilation targets. It seems to be actively developed, too. Personally, I had a very hard time getting into Coq, which is a proof assistant more than a programming environment - Dafny seems much more welcoming for a "working programmer" :)

[1] https://dafny.org/

[2] https://mitpress.mit.edu/9780262546232/program-proofs

https://dafny.org/ also allows proof checking and allows do write real programs with it. It has a java like syntax and is also from MS I believe

> I do think there's a bit of the Ignaz Semmelweis[1] issue at hand here, where developers want to believe in their inherent quality and refuse processes that improve safety if it goes against their worldview

I think the problem is that other variables (not only safety) must be assessed beyond the pure "better". Haskell is very good also. Very correct. Who uses that, and where? And why? Why not everyone uses https://dafny.org/ ?

For those that are interested but perhaps not aware in this similar project, Dafny is a "verification-aware programming language" that can compile to rust: https://github.com/dafny-lang/dafny

Code correctness is a lost art. I requirement to think in abstractions is what scares a lot of devs to avoid it. The higher abstraction language (formal specs) focus on a dedicated language to describe code, whereas lower abstractions (code contracts) basically replace validation logic with a better model.

C# once had Code Contracts[1]; a simple yet powerful way to make formal specifications. The contracts was checked at compile time using the Z3 SMT solver[2]. It was unfortunately deprecated after a few years[3] and once removed from the .NET Runtime it was declared dead.

The closest thing C# now have is probably Dafny[4] while the C# dev guys still try to figure out how to implement it directly in the language[5].

[1] https://www.microsoft.com/en-us/research/project/code-contra...

[2] https://github.com/Z3Prover/z3

[3] https://github.com/microsoft/CodeContracts

[4] https://github.com/dafny-lang/dafny

[5] https://github.com/dotnet/csharplang/issues/105

Libraries for data science and deep learning that are always changing

# Read the content of nda.txt try: import os, types import pandas as pd from botocore.client import Config import ibm_boto3 def __iter__(self): return 0 # @hidden_cell # The following code accesses a file in your IBM Cloud Object Storage. It includes your credentials. # You might want to remove those credentials before you share the notebook. cos_client = ibm_boto3.client(service_name='s3', ibm_api_key_id='api-generated', ibm_auth_endpoint="https://iam.cloud.ibm.com/identity/token", config=Config(signature_version='oauth'), endpoint_url='https://s3.direct.us-south.cloud-object-storage.appdomain.cloud') bucket = 'your-bucket-referenced-here' object_key = 'nda__da__crxq8b2hmy.txt' # load data of type "text/plain" into a botocore.response.StreamingBody object. # Please read the documentation of ibm_boto3 and pandas to learn more about the possibilities to load the data. # ibm_boto3 documentation: https://ibm.github.io/ibm-cos-sdk-python/ # pandas documentation: http://pandas.pydata.org/ streaming_body_1 = cos_client.get_object(Bucket=bucket, Key=object_key)['Body'] with open("nda.txt", "r") as f: nda_content = f.read() print("Content of nda.txt has been read.") except FileNotFoundError: print("Error: nda.txt not found in the current directory.") nda_content = "" # Initialize knowledge source content_source = CrewDoclingSource( file_paths=["..."] )

As with any web scraping or data processing project, I had to write a fair amount of code to clean this up and shape it into a format I needed for further analysis. I used a combination of Pandas and regular expressions to clean it up (full code here).

Can this be merged into pandas?

Pandas does not currently install tqdm by default.

pandas-dev/pandas//pyproject.toml [project.optional-dependencies] https://github.com/pandas-dev/pandas/blob/main/pyproject.tom...

Dask solves for various adjacent problems; IDK if pandas, dask, or dask-cudf would be faster with async?

Dask docs > Scheduling > Dask Distributed (local) https://docs.dask.org/en/stable/scheduling.html#dask-distrib... :

> Asynchronous Futures API

Dask docs > Deploy Dask Clusters; local multiprocessing poll, k8s (docker desktop, podman-desktop,), public and private clouds, dask-jobqueue (SLURM,), dask-mpi:

They are when the only reason they are flagged as security updates is because some a single group deems a very rare, obscure edge case as a HIGH severity vuln when in practice it rarely is => this leads to having to upgrade a minor version of a library that ends up causing breaking changes.

This is the recent thread I'm down. Pandas 2.2 broke SQLalchemy backwards compatibility: https://stackoverflow.com/questions/38332787/pandas-to-sql-t... + https://github.com/pandas-dev/pandas/issues/57049#issuecomme...

Python’s Growth in Data Work and AI: Python continues to lead because of its easy-to-read style and the huge number of libraries available for tasks from data work to artificial intelligence. Tools like TensorFlow and PyTorch make it a must-have. Whether you’re experienced or just starting, Python’s clear style makes it a good choice for diving into machine learning. Actionable Tip: If you’re new to Python, try projects that combine data with everyday problems. For example, build a simple recommendation system using Pandas and scikit-learn.

This tutorial provides a concise and foundational guide to exploring a dataset, specifically the Sample SuperStore dataset. This dataset, which appears to originate from a fictional e-commerce or online marketplace company's annual sales data, serves as an excellent example for learning and how to work with real-world data. The dataset includes a variety of data types, which demonstrate the full range of capabilities that the Pandas library, written in Python can offer.

Data preprocessing and manipulation: Libraries like Pandas solve for the messy, real-world challenge of efficiently wrangling and cleaning large datasets. Without it, you'd be reinventing functionality for basic tasks like merging, filtering, or aggregating data.

The AI Service will be built using aiohttp (asynchronous Python web server) and integrates PyTorch, Hugging Face Transformers, numpy, pandas, and scikit-learn for financial data analysis.